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(54) Moving object detecting method, apparatus, and computer program product 



(57) Herein disclosed a moving object detecting 
method of detecting optical flow segments indicative of 
loci of one or more moving objects apparently moved in 
a moving picture sequence including a plurality of pic- 
ture frames each indicating the one or more moving ob- 
jects at a time point to ensure that the one or more mov- 
ing objects are correctly detected, the picture frames in- 
cluding a first picture frame indicating the one or more 
moving objects at a first time point, a second picture 
frame indicating the one or more moving objects at a 
second time point, a third picture frame indicating the 
one or more moving objects at a third time point, and a 
fourth picture frame indicating the one or more moving 
objects at a fourth time point in late order, the method 
comprising the steps of: h) generating filtered optical 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 
quence in the time interval starting from the third time 
point to the first time point; and i) adding the filtered op- 
tical flow segments generated in the step (h) to the first 
optical flow segments detected in the step (e) to gener- 
ate integrated optical flow segments indicative of loci of 
the one or more moving objects apparently moved in the 
moving picture sequence in the time interval starting 
from the third time point to the first time point. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Technical Field of the Invention 

[0001] The present invention relates to a moving ob- 
ject detecting method of, a moving object detecting ap- 
paratus for, and a moving object detecting computer 
program product for detecting optical flow segments in- 
dicative of loci of one or more moving objects apparently 
moved in a moving picture sequence including a plural- 
ity of picture frames each indicating the one or more 
moving objects at a time point to ensure that the moving 
objects are correctly detected 

2. Description of the Related Art 

[0002] Up until now, there have been proposed a wide 
variety of moving object detecting methods of detecting 
optical flow segments indicative of apparent motion, i. 
e., loci of one or more moving objects apparently moved 
in a moving picture sequence such as, for example but 
not limited to, motion video pictures, including a plurality 
of picture frames each indicating the one or more mov- 
ing objects at a time point to ensure that the moving ob- 
jects are detected. 

[0003] The optical flow segments are intended to 
mean the apparent motion, i.e., loci of a moving object 
in the moving picture sequence including a plurality of 
picture frames each indicating the moving object at a 
time point as shown in FIG. 8. A moving object 810 is 
indicated by, for example, a current picture frame form- 
ing part of the moving picture sequence at a current time 
point. A moving object 820 is indicated by a past picture 
frame forming part of the moving picture sequence at a 
past time point earlier than the current time point. There 
may be provided a plurality of picture frames between 
the past picture frame and the current picture frame in 
a time interval starting from the past time point to the 
current time point. The term "current picture frame" and 
"current time point" as used herein are respectively in- 
tended to mean a picture frame later than the past pic- 
ture frame and a time point later than the past time point, 
but not limited to the present picture frame or the present 
time point in the literal sense of the words. Optical flow 
segments 830 indicate the apparent motion of the mov- 
ing object, i.e., the loci of the moving object apparently 
moved in the moving picture sequence in the time inter- 
val starting from the past time point to the current time 
point. Here, the logical addition of an area in which the 
moving object 810 occupies in the current picture frame 
and an area in which the moving object 820 occupies in 
the past picture frame will be hereinlater referred to as 
"an optical flow generating area 840". 
[0004] The optical flow segments are typically detect- 
ed and generated through the operations of: (A) dividing 
the current picture frame into a plurality of blocks each 



containing at least one picture element; (B) dividing the 
past picture frame into a plurality of blocks each con- 
taining at least one picture element; (D) sequentially 
comparing each of the blocks of the past picture frame 
s with blocks of the current picture frame to select a block 
of the current picture frame apparently placed in the vi- 
cinity of the block of the past picture frame and having 
the highest degree of similarity with the block of the past 
picture frame from among the blocks of the current pic- 
io ture frame apparently placed in the vicinity of the block 
of the past picture frame; and (D) detecting and gener- 
ating optical flow segments each starting from a block 
of the past picture frame to a block of the current picture 
frame apparently placed in the vicinity of the block of the 
is past picture frame and having the highest degree of sim- 
ilarity with the block of the past picture frame from 
among the blocks of the current picture frame apparent- 
ly placed in the vicinity of the block of the past picture 
frame. The degree of similarity may be assessed on the 

20 basis of, for example but not limited to, mean absolute 
differences (MAD), root mean square differences 
(RMSD), or correlation values. 
[0005] The aforesaid operations, however, require a 
large amount of computational processes and time. This 

25 leads to the fact that the aforesaid operations are gen- 
erally performed only in blocks of the current picture 
frame apparently placed in the vicinity of a block of the 
past picture frames by, for example but not limited to, a 
predetermined number of blocks or less, for example, a 

30 couple of blocks or less to detect and generate optical 
flow segments. This means that the operations (C) and 
(D) are performed through the operations of sequentially 
comparing each of the blocks of the past picture frame 
with the blocks of the current picture frame to select a 

35 block of the current picture frame apparently placed in 
the vicinity of the block of the past picture frame and 
having the highest degree of similarity with the block of 
the past picture frame from among the blocks of the cur- 
rent picture frame apparently placed in the vicinity of the 

40 block of the past picture frame by a predetermined 
number of blocks or less; and detecting and generating 
optical flow segments each starting from a block of the 
past picture frame to a block of the current picture frame 
apparently placed in the vicinity of the block of the past 

45 picture frame and having the highest degree of similarity 
with the block of the past picture frame from among the 
blocks of a detectable area, i.e., the blocks of the current 
picture frame apparently placed in the vicinity of the 
block of the past picture frame by a predetermined 

50 number of blocks or less. 

[0006] A part of a moving object apparently moved to, 
for instance, an undetectable area away from the de- 
tectable area, i.e., the blocks of the current picture frame 
apparently placed in the vicinity of the blocks of the past 

55 picture frame indicative of the part of the moving object 
at the past time point by more than the predetermined 
number of blocks, the aforesaid operations may inaccu- 
rately detect and generate optical flow segments each 
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starting from a block of the past picture frame indicative 
of the part of the moving object at the past time point to 
a block of the current picture frame apparently placed 
in the vicinity of the block of the past picture frame and 
having the highest degree of similarity with the block of s 
the past picture frame among the blocks of the current 
picture frame apparently placed in the vicinity of the 
block of the past picture frame by a predetermined 
number of blocks or less, which is not indicative of the 
part of the moving object. 10 
[0007] This leads to the fact that the aforesaid oper- 
ations are required to shorten the time interval starting 
from a past time point to a current time point to detect 
and generate optical flow segments indicative of appar- 
ent motion of a moving object moving at a high moving 15 
speed so that the moving object apparently moved in 
the moving picture sequence would remain in the de- 
tectable area, i.e., the blocks of the current picture frame 
at the current time point apparently placed in the vicinity 
of the block of the past picture frame at the past time 20 
point by a predetermined number of blocks or less while, 
on the other hand, the aforesaid operations are required 
to lengthen the time interval starting from a past time 
point to a current time point to detect and generate op- 
tical flow segments indicative of apparent motion of a 25 
moving object moving at a low moving speed so that the 
optical flow segments indicative of the apparent motion 
of the moving object apparently moved in the time inter- 
val would be detected and generated. 
[0008] One typical type of the conventional moving 30 
object detecting method is disclosed in Japanese Patent 
Application Laid-Open Publication No. 16776/1997. In 
order to meet the above requirements of adaptively 
changing the time interval in accordance with the mov- 
ing speed of the moving object, the aforesaid conven- 35 
tional moving object detecting method comprises the 
steps of: (a) inputting a current picture frame indicating 
a moving object at a current time point; (b) inputting a 
past picture frame indicating the moving object at a past 
time point; (c) detecting and generating optical flow seg- 40 
ments indicative of apparent motion, i.e., loci of the mov- 
ing object apparently moved from the past time point to 
the current time point; (d) evaluating the degree of va- 
lidity of the optical flow segments thus detected and gen- 
N erated; and (e) controlling the time interval starting from 45 
the past time point to the current time point on the basis 
of the result of evaluation made in the step (d). 
[0009] The aforesaid conventional moving object de- 
tecting method, however, encounters a drawback that 
the aforesaid conventional moving object detecting so 
method can detect and generate optical flow segments 
indicative of loci of a moving object moved at a constant 
moving speed while, on the other hand, the aforesaid 
conventional moving object detecting method cannot 
accurately detect and generate optical flow segments 55 
indicative of loci of a plurality of moving objects appar- 
ently moved at variable moving speeds. 
[001 0] Furthermore, the aforesaid conventional mov- 



ing object detecting method encounters another draw- 
back that the conventional moving object detecting 
method may generate false optical flow segments which 
inaccurately indicate the apparent motion, i.e., loci of a 
moving object apparently moved at a moving speed so 
high with respect to the time interval starting from a past 
time point to a current time point that the moving object 
apparently moved away from a detectable area in which 
optical flow segments can be properly detected and 
generated, to an undetectable area of the current picture 
frame apparently placed in the vicinity of the blocks of 
the past picture frame indicative of the moving object at 
the past time point by more than the predetermined 
number of blocks, resulting in the fact that the aforesaid 
conventional moving object detecting method may inac- 
curately detect and generate optical flow segments 
each starting from a block of the past picture frame in- 
dicative of a part of the moving object to a block of the 
current picture frame apparently placed in the vicinity of 
the block of the past picture frame and having the high- 
est degree of similarity with the block of the past picture 
frame among the blocks of the current picture frame ap- 
parently placed in the vicinity of the block of the past 
picture frame by a predetermined number of blocks or 
less, which is not indicative of any part of the moving 
object as shown in FIG. 9. In FIG. 9, a moving object 
91 0 indicated by, for example, a current picture frame is 
apparently moved from a moving object 920 indicated 
by a past picture frame earlier than the current picture 
frame. The logical addition of an area in which the mov- 
ing object 91 0 occupies in the current picture frame and 
an area in which the moving object 920 occupies in the 
past picture frame is an optical flow generating area 940 . 
Assuming that the moving object 91 0 is placed in an un- 
detectable area in which optical flow segments cannot 
be properly detected and generated, the optical flow 
generating area 940 may contain an area in which false 
optical flow segments may be detected and generated, 
which will be hereinlater referred to as "false optical flow 
generating area 930" in which false optical flow seg- 
ments inaccurately indicating loci of a moving object are 
detected and generated. 

[0011] The present invention is made with a view to 
overcoming the previously mentioned drawbacks inher- 
ent to the conventional moving object detecting method 
and the conventional moving object detecting appara- 
tus. 

SUMMARY OF THE INVENTION 

[001 2] It is therefore an object of the present invention 
to provide a moving object detecting method which can 
detect optical flow segments indicative of loci of one or 
more moving objects apparently moved at variable mov- 
ing speeds in a moving picture sequence including a plu- 
rality of picture frames each indicating the one or more 
moving objects at a time point to ensure that the moving 
objects are correctly detected. 
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[0013] It is another object of the present invention to 
provide a moving object detecting method which can ac- 
curately detect optical flow segments indicative of loci 
of one or more moving objects apparently moved at var- 
iable moving speeds in a moving picture sequence in- 
cluding a plurality of picture frames each indicating the 
one or more moving objects at a time point by eliminat- 
ing false optical flow segments which inaccurately indi- 
cate loci of a moving object apparently moved at a mov- 
ing speed so high that the moving object apparently 
moved to an undetectable area away from a detectable 
area in which optical flow segments can be properly de- 
tected and generated to ensure that the moving objects 
are correctly detected. 

[0014] It is a further object of the present invention to 
provide a moving object detecting apparatus which can 
detect optical flow segments indicative of loci of one or 
more moving objects apparently moved at variable mov- 
ingspeeds in a moving picture sequence including a plu- 
rality of picture frames each indicating the one or more 
moving objects at a time point to ensure that the moving 
objects are correctly detected. 
[001 5] It is a still further object of the present invention 
to provide a moving object detecting apparatus which 
can accurately detect optical flow segments indicative 
of loci of one or more moving objects apparently moved 
at variable moving speeds in a moving picture sequence 
including a plurality of picture frames each indicating the 
one or more moving objects at a time point by eliminat- 
ing false optical flow segments which inaccurately indi- 
cate loci of a moving object apparently moved at a mov- 
ing speed so high that the moving object apparently 
moved to an undetectable area away from the detecta- 
ble area in which optical flow segments can be property 
detected and generated to ensure that the moving ob- 
jects are correctly detected. 

[001 6] It is a yet further object of the present invention 
to provide a moving object detecting computer program 
product which can detect optical flow segments indica- 
tive of loci of one or more moving objects apparently 
moved at variable moving speeds in a moving picture 
sequence including a plurality of picture frames each in- 
dicating the one or more moving objects at a time point 
to ensure that the moving objects are correctly detected. 
[0017] It is a yet further object of the present invention 
to provide a moving object detecting computer program 
product which can accurately detect optical flow seg- 
ments indicative of loci of one or more moving objects 
apparently moved at variable moving speeds in a mov- 
ing picture sequence including a plurality of picture 
frames each indicating the one or more moving objects 
at a time point by eliminating false optical flow segments 
which inaccurately indicate loci of a moving object ap- 
parently moved at a moving speed so high that the mov- 
ing object apparently moved to an undetectable area 
away from the detectable area in which optical flow seg- 
ments can be property detected and generated to en- 
sure that the moving objects are correctly detected. 



[001 8] In accordance with a first aspect of the present 
invention, there is provides a moving object detecting 
method of detecting optical flow segments indicative of 
loci of one or more moving objects apparently moved in 
5 a moving picture sequence including a plurality of pic- 
ture frames each indicating the one or more moving ob- 
jects at a time point to ensure that the one or more mov- 
ing objects are correctly detected, the picture frames in- 
cluding a first picture frame indicating the one or more 

10 moving objects at a first time point, a second picture 
frame indicating the one or more moving objects at a 
second time point, a third picture frame indicating the 
one or more moving objects at a third time point, and a 
fourth picture frame indicating the one or more moving 

is objects at a fourth time point in late order, the method 
comprising the steps of: a) inputting the first picture 
frame; b) inputting the second picture frame; c) inputting 
the third picture frame; d) inputting the fourth picture 
frame; e) detecting first optical flow segments indicative 

20 of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 
starting from the second time point to the first time point 
on the basis of the first picture frame inputted in the step 
(a) and the second picture frame inputted in the step (b); 

25 f) detecting second optical flow segments indicative of 
loci of the one or more moving objects apparently moved 
in the moving picture sequence in a time interval starting 
from the third time point to the first time point on the basis 
of the first picture frame inputted in the step (a) and the 

30 third picture frame inputted in the step (c); g) detecting 
third optical flow segments indicative of loci of the one 
or more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the 
fourth time point to the third time point on the basis of 

35 the third picture frame inputted in the step (c) and the 
fourth picture frame inputted in the step (d); h) subtract- 
ing the third optical flow segments detected in the step 
(g) from the second optical flow segments detected in 
the step (0 to generate filtered optical flow segments 
indicative of loci of the one or more moving objects ap- 
parently moved in the moving picture sequence in the 
time interval starting from the third time point to the first 
time point; and i) adding the filtered optical flow seg- 
ments generated in the step (h) to the first optical flow 

45 segments detected in the step (e) to generate integrated 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in the time interval starting from the 
third time point to the first time point. 

so [0019] In the aforesaid step (e) may have the steps 
of: (e1) dividing the first picture frame inputted in the 
step (a) into a plurality of blocks each containing at least 
one picture element; (e2) dividing the second picture 
frame inputted in the step (b) into a plurality of blocks 

55 each containing at least one picture element; (e3) se- 
quentially comparing each of the blocks of the second 
picture frame divided in the step (e2) with blocks of the 
first picture frame divided in the step (e1) to select a 
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block of the first picture frame apparently placed in the 
vicinity of the block of the second picture frame and hav- 
ing the highest degree of similarity with the block of the 
second picture frame from among the blocks of the first 
picture frame apparently placed in the vicinity of the 
block of the second picture frame; and (e4) detecting 
optical flow segments each starting from a block of the 
second picture frame divided in the step (e2) to a block 
of the first picture frame selected in the step (e3) appar- 
ently placed in the vicinity of the block of the second 
picture frame and having the highest degree of similarity 
with the block of the second picture frame from among 
the blocks of the first picture frame apparently placed in 
the vicinity of the block of the second picture frame. 
[0020] In accordance with a second aspect of the 
present invention, there is provided a moving object de- 
tecting method may further comprising the steps of: j) 
detecting neighboring areas neighboring to optical flow 
generating areas in which the integrated optical flow 
segments are detected in the step (i) to obtain neighbor- 
ing area information; k) estimating moving object candi- 
date areas in the moving picture sequence in which the 
moving objects are supposed to be present on the basis 
of the neighboring area information detected in the step 
(j) to obtain moving object candidate area information in 
the moving picture sequence; and 1 ) detecting the mov- 
ing objects on the basis of the moving object candidate 
area information obtained in the step (k). The aforesaid 
moving object candidate area information may include 
information about the sizes, shapes, positions, picture 
elements, optical flow information of the moving object 
candidate areas. Alternatively, the aforesaid moving ob- 
ject detecting method may comprise the steps of: a2) 
inputting the first picture frame; b2) inputting the second 
picture frame; c2) inputting the third picture frame; d2) 
inputting the fourth picture frame; a3) storing the first 
picture frame; b3) storing the second picture frame; c3) 
storing the third picture frame; d3) storing the fourth pic- 
ture frame; e1) detecting first optical flow segments in- 
dicative of loci of the one or more moving objects appar- 
ently moved in the moving picture sequence in a time 
interval starting from the second time point to the first 
time point on the basis of the first picture frame stored 
in the step (a2) and the second picture frame stored in 
the step (b2); f) detecting second optical flow segments 
indicative of loci of the one or more moving objects ap- 
parently moved in the moving picture sequence in a time 
interval starting from the third time point to the first time 
point on the basis of the first picture frame stored in the 
step (a2) and the third picture frame stored in the step 
(c2); g) detecting third optical flow segments indicative 
of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 
starting from the fourth time point to the third time point 
on the basis of the third picture frame stored in the step 
(c2) and the fourth picture frame stored in the step (d2); 
h) subtracting the third optical flow segments detected 
in the step (g) from the second optical flow segments 



detected in the step (f) to generate filtered optical flow 
segments indicative of loci of the one or more moving 
objects apparently moved in the moving picture se- 
quence in the time interval starting from the third time 
5 point to the first time point; i) adding the filtered optical 
flow segments generated in the step (h) to the first op- 
tical flow segments detected in the step (e) to generate 
integrated optical flow segments indicative of loci of the 
one or more moving objects apparently moved in the 
io moving picture sequence in the time interval starting 
from the third time point to the first time point; j) detecting 
neighboring areas neighboring to optical flow generat- 
ing areas in which the integrated optical flow segments 
are detected in the step (i) to obtain neighboring area 
*5 information in the moving picture sequence; k) estimat- 
ing moving object candidate areas in the moving picture 
sequence in which the moving objects are supposed to 
be present on the basis of the neighboring area infor- 
mation detected in the step (j) to obtain moving object 
20 candidate area information in the moving picture se- 
quence; and 1 ) detecting the moving objects in the mov- 
ing picture sequence on the basis of the moving object 
candidate area information obtained in the step (k). The 
aforesaid moving object detecting method may further 
25 comprise the steps of: m) repeating the steps (a2) to (1 ), 
and tracking the moving objects in the moving picture 
sequence detected in the step (1) on the basis of the 
moving object candidate area information in the moving 
picture sequence accumulatively obtained in the step (k) 
30 to estimate moving object information including the 
number of the moving objects and the amount of appar- 
ent motion made by the moving objects; and n) control- 
ling the time interval between the first time point and the 
second time point, the time interval between the second 
35 time point and the third time point, and the time interval 
between the third time point and the fourth time point, 
and the number of the integrated optical flow segments 
on the basis of the moving object information estimated 
in the step (m). 
40 [0021 ] In accordance with a third aspect of the present 
invention, there is provided a moving object detecting 
apparatus for detecting optical flow segments indicative 
of loci of one or more moving objects apparently moved 
in a moving picture sequence including a plurality of pic- 
45 ture frames each indicating the one or more moving ob- 
jects at a time point to ensure that the one or more mov- 
ing objects are correctly detected, the picture frames in- 
cluding a first picture frame indicating the one or more 
moving objects at a first time point, a second picture 
50 frame indicating the one or more moving objects at a 
second time point, a third picture frame indicating the 
one or more moving objects at a third time point, and a 
fourth picture frame indicating the one or more moving 
objects at a fourth time point in late order, the moving 
55 object detecting apparatus comprising: a picture input- 
ting section for inputting the moving picture sequence 
including the first picture frame, the second picture 
frame, the third picture frame, and the fourth picture 
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i 

frame; a picture storage section for storing the moving 
picture sequence including the first picture frame, the 
second picture frame, the third picture frame, and the 
fourth picture frame inputted by the picture inputting sec- 
tion; and an optical flow detecting section for detecting 5 
integrated optical flow segments on the basis of the first 
picture frame inputted by the picture inputting section 
and the second picture frame, the third picture frame, 
and the fourth picture frame stored in the picture storage 
section. w 
[0022] The aforesaid optical flow detecting section 
may includes: a first optical flow detecting portion for de- 
tecting first optical flow segments indicative of loci of the 
one or more moving objects apparently moved in the 
moving picture sequence in a time interval starting from is 
the second time point to the first time point on the basis 
of the first picture frame inputted by the picture inputting 
section and the second picture frame stored in the pic- 
ture storage section; a second optical flow detecting por- 
tion for detecting second optical flow segments indica- 20 
tive of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 
starting from the third time point to the first time point on 
the basis of the first picture frame inputted by the picture 
inputting section and the third picture frame stored in the 25 
picture storage section; a false optical flow detecting 
portion for detecting third optical flow segments indica- 
tive of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 
starting from the fourth time point to the third time point 30 
on the basis of the third picture frame stored in the pic- 
ture storage section and the fourth picture frame stored 
in the picture storage section; an optical flow subtracting 
portion for subtracting the third optical flow segments 
detected by the false optical flow detecting portion from 35 
the second optical flow segments detected by the sec- 
ond optical flow detecting portion to generate filtered op- 
tical flow segments indicative of loci of the one or more 
moving objects apparently moved in the moving picture 
sequence in the time interval starting from the third time 40 
point to the first time point; and an optical flow adding 
portion for adding the filtered optical flow segments gen- 
erated by the optical flow subtracting portion to the first 
optical flow segments detected by the first optical flow 
detecting portion to generate integrated optical flow seg- 45 
ments indicative of loci of the one or more moving ob- 
jects apparently moved in the moving picture sequence 
in the time interval starting from the third time point to 
the first time point. 

[0023] Alternatively, the aforesaid moving object de- so 
tecting apparatus may further comprising an infrared 
camera unit for outputting a moving picture sequence of 
a heat distribution picture sequence including a plurality 
of picture frames including a first picture frame indicating 
the one or more moving objects at a first time point, a 55 
second picture frame indicating the one or more moving 
objects at a second time point, a third picture frame in- 
dicating the one or more moving objects at a third time 



point, and a fourth picture frame indicating the one or 
more moving objects at a fourth time point in late order. 
The picture inputting section may be replaced with a 
heat distribution picture inputting section for inputting 
the first picture frame, the second picture frame, the third 
frame picture, and the fourth frame picture forming part 
of the moving picture sequence of a heat distribution pic- 
ture sequence outputted by the infrared camera unit; 
and the picture storage section may be replaced with a 
heat distribution picture storage section for storing the 
first picture frame, the second picture frame, the third 
frame picture, and the fourth frame picture forming part 
of the moving picture sequence of a heat distribution pic- 
ture sequence inputted by the heat distribution picture 
inputting section. 

[0024] In accordance with a fourth aspect of the 
present invention, there is provided a moving object de- 
tecting program product comprising a computer usable 
storage medium having computer readable code encod- 
ed herein for detecting optical flow segments indicative 
of loci of one or more moving objects apparently moved 
in a moving picture sequence including a plurality of pic- 
ture frames each indicating the one or more moving ob- 
jects at a time point to ensure that the one or more mov- 
ing objects are correctly detected the picture frames in- 
cluding a first picture frame indicating the one or more 
moving objects at a first time point, a second picture 
frame indicating the one or more moving objects at a 
second time point, a third picture frame indicating the 
one or more moving objects at a third time point, and a 
fourth picture frame indicating the one or more moving 
objects at a fourth time point in late order, the computer 
program product comprising: (a) computer readable 
program code for inputting the first picture frame; (b) 
computer readable program code for inputting the sec- 
ond picture frame; (c) computer readable program code 
for inputting the third picture frame; (d) computer read- 
able program code for inputting the fourth picture frame; 
(e) computer readable program code for detecting first 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the sec- 
ond time point to the first time point on the basis of the 
first picture frame inputted by the computer readable 
program code (a) and the second picture frame inputted 
by the computer readable program code (b); (f)compu- 
ter readable program code for detecting second optical 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 
quence in a time interval starting from the third time point 
to the first time point on the basis of the first picture 
frame inputted by the computer readable program code 
(a) and the third picture frame inputted by the computer 
readable program code (c); (g) detecting third optical 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 
quence in a time interval starting from the fourth time 
point to the third time point on the basis of the third pic- 
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ture frame inputted by the computer readable program 
code (c) and the fourth picture frame inputted by the 
computer readable program code (d); (h) computer 
readable program code for subtracting the third optical 
flow segments detected by the computer readable pro- 5 
gram code (g) from the second optical flow segments 
detected by the computer readable program code (f) to 
generate filtered optical flow segments indicative of loci 
of the one or more moving objects apparently moved in 
the moving picture sequence in the time interval starting io 
from the third time point to the first time point; and (i) 
computer readable program code for adding the filtered 
optical flow segments generated by the computer read- 
able program code (h) to the first optical flow segments 
detected by the computer readable program code (e) to 15 
generate integrated optical flow segments indicative of 
loci of the one or more moving objects apparently moved 
in the moving picture sequence in the time interval start- 
ing from the third time point to the first time point. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0025] The objects, features and advantages of the 
present invention will become apparent as the descrip- 
tion proceeds when taken in conjunction with the ac- 25 
companying drawings, in which: 

FIG. 1 is a flowchart showing a first embodiment of 
the moving object detecting method according to 
the present invention; 30 
FIG. 2 is a schematic diagram explaining the proc- 
esses performed by the first embodiment of the 
moving object detecting method shown in FIG. 1; 
FIG. 3 is a flowchart showing a second embodiment 
of the moving object detecting method according to 35 
the present invention; 

FIG. 4 is a block diagram showing a third embodi- 
ment of the moving object detecting apparatus ac- 
cording to the present invention; 
FIG. 5 is a flowchart showing a fourth embodiment 40 
of the moving object detecting method according to 
the present invention; 

FIG. 6 is a block diagram showing a fifth embodi- 
ment of the moving object detecting apparatus ac- 
cording to the present invention; 45 
FIG. 7 is a block diagram showing a sixth embodi- 
ment of the moving object detecting apparatus ac- 
cording to the present invention; 
FIGS. 8A and 8 B are a set of schematic diagrams 
briefly explaining the outline of optical flow seg- so 
ments; 

FIG. 9 is a schematic diagram explaining false op- 
tical flow segments which inaccurately Indicate loci 
of a moving object apparently moved at an exces- 
sively high moving speed with respect to the time 55 
interval starting from a past time point to a current 
time point; 

FIG. 10 is a schematic diagram explaining a princi- 



ple of eliminating false optical flow segments inac- 
curately indicating loci of a moving object; and 
FIG. 1 1 is a schematic diagram explaining the proc- 
esses of eliminating the false optical flow segments. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0026] The preferred embodiments of the moving ob- 
ject detecting method and the moving object detecting 
apparatus will be described with reference to FIGS. 1 to 
11 . Throughout the following detailed description, simi- 
lar reference characters and numbers refer to similar el- 
ements in all figures of the drawings. 
[0027] Referring now to the drawings, in particular 
FIGS. 1 and 2, there is shown a first preferred embodi- 
ment of the moving object detecting method according 
to the present invention of detecting optical flow seg- 
ments indicative of loci of one or more moving objects 
apparently moved in a moving picture sequence includ- 
ing a plurality of picture frames to ensure that the one 
or more moving objects are correctly detected. The 
moving picture sequence may be, for example but not 
limited to, a motion video picture, and include a plurality 
of picture frames each indicating the one or more mov- 
ing objects at a time point. The picture frames include a 
first picture frame indicating the one or more moving ob- 
jects at a first time point, a second picture frame indicat- 
ing the one or more moving objects at a second time 
point, a third picture frame indicating the one or more 
moving objects at a third time point, and a fourth picture 
frame indicating the one or more moving objects at a 
fourth time point in late order as shown in FIG. 2. This 
means that the fourth picture frame is older than the third 
picture frame. The third picture frame is older than the 
second picture frame. The second picture frame is older 
than the first picture frame. The first picture frame may 
be, for example but not limited to a current picture frame. 
There may be provided a plurality of picture frames be- 
tween the first picture frame and the second picture 
frame in a time interval starting from the first time point 
to the second time point. There may be provided a plu- 
rality of picture frames between the second picture 
frame and the third picture frame in a time interval start- 
ing from the second time point to the third time point. 
There may be provided a plurality of picture frames be- 
tween the third picture frame and the fourth picture 
frame in a time interval starting from the third time point 
to the fourth time point. 

[0028] The optical flow segments are intended to 
mean the apparent motion, i.e., loci of a moving object 
in the moving picture sequence including a plurality of 
picture frames each indicating the moving object at a 
time point as shown in FIG. 8. A moving object 810 is 
indicated by a first picture frame, for example, a current 
picture frame forming part of the moving picture se- 
quence at a first time point, for example, a current time 
point. A moving object 820 is indicated by a second pic- 
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ture frame forming part of the moving picture sequence 
at a second time point later than the first time point. 
There may be provided a plurality of picture frames be- 
tween the second picture frame and the first picture 
frame in a time interval starting from the second time 5 
point to the first time point. Optical flow segments 830 
indicate the apparent motion of the moving object, i.e., 
the loci of the moving object apparently moved in the 
moving picture sequence in the time interval starting 
from the second time point to the first time point. Here, 10 
a logical addition of an area in which the moving object 
810 occupies in the current picture frame and an area 
in which the moving object 820 occupies in the past pic- 
ture frame will be hereinlater referred to as "an optical 
flow generating area 840". is 
[0029] The flowchart of the first embodiment of the 
moving object detecting method of detecting optical flow 
segments indicative of loci of one or more moving ob- 
jects apparently moved in a moving picture sequence 
including a plurality of picture frames each indicating the 20 
one or more moving objects at a time point to ensure 
that the one or more moving objects are correctly de- 
tected is shown in FIG. 1. The first embodiment of the 
moving object detecting method according to the 
present invention will be described with reference to 25 
FIG. 1. 

[0030] In the step S110, the first picture frame is in- 
putted. The step S110 goes forward to the step S121, 
in which the second picture frame is inputted. The step 
S121 goesforwardtothestepS131,inwhichfirstoptical 30 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 
quence in a time interval starting from the second time 
point to the first time point is detected on the basis of 
the first picture frame inputted in the step S1 1 0 and the 35 
second picture frame inputted in the step S121 . 
[0031 ] The optical flow segments are typically detect- 
ed through the operations of: 

(e1) dividing the first picture frame inputted in the 40 
step S110 into a plurality of blocks each containing 
at least one picture element; 
(e2) dividing the second picture frame inputted in 
the step S121 into a plurality of blocks each con- 
taining at least one picture element; 45 
(e3) sequentially comparing each of the blocks of 
the second picture frame divided in the step (e2) 
with blocks of the first picture frame divided in the 
step S121 to select a block of the first picture frame 
apparently placed in the vicinity of the block of the so 
second picture frame and having the highest de- 
gree of similarity with the block of the second picture 
frame from among the blocks of the first picture 
frame apparently placed in the vicinity of the block 
of the second picture frame; and 55 
(e4) detecting optical flow segments each starting 
from a block of the second picture frame divided in 
the step (e2) to a block of the first picture frame se- 



lected in the step (e3) apparently placed in the vi- 
cinity of the block of the second picture frame and 
having the highest degree of similarity with the block 
of the second picture frame from among the blocks 
of the first picture frame apparently placed in the 
vicinity of the block of the second picture frame. 

[0032] The degree of similarity may be assessed on 
the basis of, for example but not limited to, mean abso- 
lute differences (MAD), root mean square differences 
(RMSD), or correlation values. 

[0033] In the step S122, the second picture frame is 
inputted. The step S122 goes forward to the step S132, 
in which second optical flow segments indicative of loci 
of the one or more moving objects apparently moved in 
the moving picture sequence in a time interval starting 
from the third time point to the first time point is detected 
on the basis of the first picture frame inputted in the step 
S110 and the third picture frame inputted in the step S 
1 22 in a similar manner as described above. 
[0034] The time interval starting from the second time 
point to the first time point is shorter than the time inter- 
val starting from the third time point to the first time point. 
This means that the first optical flow segments detected 
in the step S1 31 indicate loci of the one or more moving 
objects apparently moved at a high moving speed in the 
moving picture sequence in the time interval starting 
from the second time point to the first time point while, 
on the other hand, the second optical flow segments de- 
tected in the step S132 indicate loci of the one or more 
moving objects apparently moved at a low moving 
speed in the moving picture sequence in the time inter- 
val starting from the third time point to the first time point 
[0035] The second optical flow segments detected in 
the step S132, however, may include false optical flow 
segments inaccurately indicate loci of a moving object 
apparently moved at a moving speed so high that the 
moving object apparently moved in the time interval 
starting from the third time point to the first time point to 
an undetectable area away from the detectable area in 
which optical flow segments can be properly detected 
and generated as shown in FIG. 9. In FIG. 9, a moving 
object 910 indicated by, for example, a current picture 
frame is apparently moved from a moving object 920 
indicated by a past picture frame earlier than the current 
picture frame. The logical addition of an area in which 
the moving object 910 occupies in the current picture 
frame and an area in which the moving object 820 oc- 
cupies in the past picture frame is an optical flow gen- 
erating area 940. Assuming that the moving object 91 0 
is placed in the undetectable area in which optical flow 
segment cannot be property detected and generated, 
the optical flow generating area 940 may contain a false 
optical flow generating area 930 in which in which false 
optical flow segments inaccurately indicating loci of a 
moving object are detected. 

[0036] In order to eliminate the false optical flow seg- 
ments, the moving object detecting method according 
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to the present invention further comprises the steps 
S123, S133, and S142. In the step S123, the fourth pic- 
ture frame is inputted. The step S123 goes forward to 
the step S133 in which the third optical flow segments 
indicative of loci of the one or more moving objects ap- 
parently moved in the moving picture sequence in a time 
interval starting from the fourth time point to the third 
time point is detected on the basis of the third picture 
frame inputted in the step S122 and the fourth picture 
frame inputted in the step S123. 
[0037] The principle of eliminating the false optical 
flow segments will be described in detail with reference 
to FIGS. 10 and 11. In FIG. 10, pictures designated by 
1 01 0 and 1 030 indicate an apparent motion of a moving 
object moved at a high moving speed and an optical flow 
generating area 940 generated on the basis of picture 
frames in a long time interval, while, on the other hand, 
pictures designated by 1020 and 1040 indicate an ap- 
parent motion of the moving object moved at the high 
speed and an optical flow generating area 931 generat- 
ed on the basis of picture frames inputted in a short time 
interval. The moving object apparently moved from an 
area 930 to an area 941 . Assuming that the area 941 is 
an undetectable area with respect to the area 930 in 
which optical flow segments cannot be properly detect- 
ed and generated, the optical flow generating area 940 
contains a false optical flow generating area 930 in 
which false optical flow segments inaccurately indicat- 
ing loci of the moving object are detected and generat- 
ed. In order to eliminate the false optical flow segments, 
the optical flow generating area 931 generated on the 
basis of picture frames inputted in a short time interval 
is subtracted from the optical flow generating area 940 
generated on the basis of picture frames inputted in a 
long time interval to generate a new optical flow gener- 
ating area 941 , which is supposed to hardly contain the 
false optical flow segments as shown in a picture des- 
ignated by 1 050. In FIG . 1 1 , pictures designated by 1 1 1 0 
and 1 1 30 indicate an apparent motion of a moving object 
moved at a low moving speed and an optical flow gen- 
erating area generated on the basis of picture frames in 
a long time interval. Pictures designated by 1120 and 
1140 are supposed to indicate an apparent motion of 
the moving object moved at the low speed and an optical 
flow generating area generated on the basis of picture 
frames inputted in a short time interval. The moving ob- 
ject apparently moved at a very low speed with respect 
to the short time interval so that no optical flow genera- 
tion area, however, is generated in the picture designat- 
ed by 1 1 40. The moving object apparentry moved a very 
short distance as shown in the pictures designated by 
1110 and 1130. In this case, no optical flow generating 
area is subtracted from the optical flow generating area 
generated on the basis of picture frames inputted in a 
long time interval. 

[0038] This leads to the fact that the third optical flow 
segments indicative of loci of the one or more moving 
objects apparentry moved in a short time interval starting 



from the fourth time point to the third time point detected 
in the step S133 are to be subtracted from the second 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
5 picture sequence in a long time interval starting from the 
third time point to the first time point detected in the step 
S 132 in order to eliminate the false optical flow segment 
generating area. 

[0039] The step S1 33 goes forward to the step S1 42 
10 in which the third optical flow segments detected in the 
step S133 from the second optical flow segments de- 
tected in the step S 132 to generate filtered optical flow 
segments indicative of loci of the one or more moving 
objects apparentry moved in the moving picture se- 
15 quence in the time interval starting from the third time 
point to the first time point. 

[0040] The step S1 42 goes forward to the step S1 50 
in which the filtered optical flow segments generated in 
the step S142 are added to the first optical flow seg- 

20 ments detected in the step S1 31 to generate integrated 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in the time interval starting from the 
third time point to the first time point, which contain false 

25 less optical flow segments. 

[0041] As described earlier, the first optical flow seg- 
ments detected in the step S131 indicate loci of the one 
or more moving objects apparently moved at a high 
moving speed in the moving picture sequence in the 

30 time interval starting from the second time point to the 
first time point while, on the other hand, the filtered op- 
tical flow segments generated in the step S1 42 from the 
second optical flow segments detected in the step S132 
indicate loci of the one or more moving objects appar- 

35 entfy moved at a low moving speed in the moving picture 
sequence in the time interval starting from the third time 
point to the first time point. This means that the integrat- 
ed optical flow segments thus generated can indicate 
loci of one or more moving objects apparently moved at 

40 a high moving speed in the moving picture sequence in 
the time interval starting from the second time point to 
the first time point, and loci of one or more moving ob- 
jects apparently moved at a low moving speed in the 
moving picture sequence in the time interval starting 

45 from the third time point to the first time point. 

[0042] From the foregoing description, it is to be un- 
derstood that the first embodiment of the moving object 
detecting method according to the present invention can 
detect optical flow segments indicative of loci of one or 

50 more moving objects apparently moved at variable mov- 
ing speeds in a moving picture sequence including a plu- 
rality of picture frames each indicating the one or more 
moving objects at a time point to ensure that the moving 
objects are correctly detected. 

55 [0043] Furthermore, the first embodiment of the mov- 
ing object detecting method according to the present in- 
vention can accurately detect optical flow segments in- 
dicative of loci of one or more moving objects apparently 
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moved at variable moving speeds in a moving picture 
sequence including a plurality of picture frames each in- 
dicating the one or more moving objects at a time point 
by eliminating false optical flow segments which inac- 
curately indicate loci of a moving object apparently 5 
moved at a moving speed so high that the moving object 
apparently moved to an undetectable area away from 
the detectable area in which optical flow segments can 
be properly detected and generated to ensure that the 
moving objects are correctly detected. 10 
[0044] Referring then to FIG. 3 of the drawings, there 
is shown a second preferred embodiment of the moving 
object detecting method according to the present inven- 
tion. The second embodiment of the moving object de- 
tecting method is similar to the first embodiment of the 15 
moving object detecting method except for the fact that 
the second embodiment of the moving object detecting 
method further comprises the step of detecting the mov- 
ing objects. 

[0045] The second embodiment of the moving object 20 
detecting method according to the present invention will 
be described with reference to FIG. 3. 
[0046] In the step S110, the first picture frame is in- 
putted. The step S11 0 goes forward to the step S121 , 
in which the second picture frame is inputted. The step 25 
S 1 21 goes forward to the step S1 31 , in which first optical 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 
quence in a time interval starting from the second time 
point to the first time point is detected on the basis of 30 
the first picture frame inputted in the step S1 1 0 and the 
second picture frame inputted in the step S121 . In the 
step S122, the second picture frame is inputted. The 
step S122 goes forward to the step S132, in which sec- 
ond optical flow segments indicative of loci of the one 35 
or more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the third 
time point to the first time point is detected on the basis 
of the first picture frame inputted in the step S110 and 
the third picture frame inputted in the step S1 22 in a sim- 40 
ilar manner as described above. 
[0047] In the step S1 23, the fourth picture frame is in- 
putted. The step S1 23 goes forward to the step S1 33 in 
which the third optical flow segments indicative of loci 
of the one or more moving objects apparently moved in 45 
the moving picture sequence in a time interval starting 
from the fourth time point to the third time point is de- 
tected on the basis of the third picture frame inputted in 
the step S122 and the fourth picture frame inputted in 
the step S1 23. The step S1 33 goes forward to the step so 
S142 in which the third optical flow segments detected 
in the step S1 33 from the second optical flow segments 
detected in the step S132 to generate filtered optical 
flow segments indicative of loci of the one or more mov- 
ing objects apparently moved in the moving picture se- 55 
quence in the time interval starting from the third time 
point to the first time point. 

[0048] The step S142 goes forward to the step S150 



in which the filtered optical flow segments generated in 
the step S142 are added to the first optical flow seg- 
ments detected in the step S131 to generate integrated 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in the time interval starting from the 
third time point to the first time point, which contain false 
less optical flow segments. 

[0049] The second embodiment of the moving object 
detecting method according to the present invention fur- 
ther comprises the step S320. In the step S320, the 
moving objects are detected on the basis of the integrat- 
ed optical flow segments generated in the step S150. 
More specifically, in the step S320, firstly neighboring 
areas neighboring to optical flow generating areas in 
which the integrated optical flow segments are detected 
in the step S1 50 are detected to obtain neighboring area 
information in the moving picture sequence; secondly, 
moving object candidate areas in the moving picture se- 
quence in which the moving objects are supposed to be 
present are estimated on the basis of the neighboring 
area information thus detected to obtain moving object 
candidate area information in the moving picture se- 
quence; and thirdly, the moving objects are detected on 
the basis of the moving object candidate area informa- 
tion thus obtained. 

[0050] Then, the step S320 goes forward to the step 
S330, in which it is judged whether the moving object 
detecting process will be completed or not. If it is judged 
by, for example but not limited to, an operator, that the 
detecting process will be completed, the step S330 goes 
to END. Otherwise, the step S330 goes back to the step 
S110 and the moving object detecting process will con- 
tinue with a first picture frame, a second picture frame, 
a third picture frame, and a fourth picture frame newly 
inputted. 

[0051] Neighboring area information herein used is in- 
tended to mean information about neighboring areas 
neighboring to optical flow generating areas in which the 
integrated optical flow segments are detected, and may 
include, for example but not limited to, picture element 
information about the neighboring areas, moving object 
candidate area information herein used is intended to 
mean information about moving object candidate area 
in which the moving objects are supposed to be present, 
and may include, for example but not limited to, picture 
element information about the moving object candidate 
area. 

[0052] This means that the moving object candidate 
areas in which the moving objects are supposed to be 
present may be estimated on the basis of the neighbor- 
ing area information to obtain moving object candidate 
area information. The neighboring area information may 
include information about, for example, picture element 
information about the neighboring areas neighboring to 
the optical flow generating areas in which the integrated 
optical flow segments are detected. The moving object 
candidate area information may include information 
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about, for example but not limited to, the sizes, shapes, 
positions, picture elements or optical flow information of 
the moving object candidate areas. Then, the moving 
objects are detected on the basis of the moving object 
candidate area information thus obtained. The moving s 
objects may be detected on the basis of, for example 
but not limited to, sizes, shapes, positions, picture ele- 
ments or optical flow information of the moving object 
candidate areas, or any combination thereof. The opti- 
cal flow information may include, for example but not to 
limited to, sizes, directions, positions, and the degrees 
of similarity of the optical flow segments. 
[0053] From the foregoing description, it is to be un- 
derstood that the second embodiment of the moving ob- 
ject according to the present invention can detect optical *5 
flow segments indicative of loci of one or more moving 
objects apparently moved at variable moving speeds in 
a moving picture sequence including a plurality of pic- 
ture frames each indicating the one or more moving ob- 
jects at a time point to ensure that the moving objects 20 
are correctly detected. 

[0054] Furthermore, the second embodiment of the 
moving object detecting method according to the 
present invention can accurately detect optical flow seg- 
ments indicative of loci of one or more moving objects 25 
apparently moved at variable moving speeds in a mov- 
ing picture sequence including a plurality of picture 
frames each indicating the one or more moving objects 
at a time point by eliminating false optical flow segments 
which inaccurately indicate loci of a moving object ap- 30 
parently moved at a moving speed so high that the mov- 
ing object apparently moved to an undetectable area 
away from the detectable area in which optical flow seg- 
ments can be properly detected and generated to en- 
sure that the moving objects are correctly detected. 35 
[0055] Referring to FIG.. 4 of the drawings, there is 
shown a third preferred embodiment of the moving ob- 
ject detecting apparatus 400 according to the present 
invention. The third embodiment of the moving object 
detecting apparatus 400 is shown in FIG. 4 as compris- 40 
ing: a camera unit 500, a picture inputting section 410, 
a picture storage section 420, and an optical flow de- 
tecting section 430. The camera unit 500 is adapted to 
convert optical moving picture information into digital 
moving picture information to output moving picture se- 4s 
quence including a first picture frame indicating the one 
or more moving objects at a first time point, a second 
picture frame indicating the one or more moving objects 
at a second time point, a third picture .frame indicating 
the one or more moving objects at a third time point, and so 
a fourth picture frame indicating the one or more moving 
objects at a fourth time point in late order. 
[0056] The picture inputting section 41 0 is adapted to 
input the moving picture sequence including the first pic- 
ture frame, the second picture frame, the third picture 55 
frame, and the fourth picture frame. The picture storage 
section 420 is adapted to store the moving picture se- 
quence including the first picture frame, the second pic- 



ture frame, the third picture frame, and the fourth picture 
frame inputted by the picture inputting section 41 0. 
[0057] The optical flow detecting section 430 is adapt- 
ed to detect integrated optical flow segments on the ba- 
sis of the first picture frame inputted by the picture in- 
putting section 410 and the second picture frame, the 
third picture frame, and the fourth picture frame stored 
in the picture storage section 420 by performing the first 
embodiment of the moving object detecting method as 
described earlier. 

[0058] This means that the optical flow detecting sec- 
tion 430 includes: a first optical flow detecting portion 
(not shown), a second optical flow detecting portion (not 
shown), a false optical flow detecting portion (not 
shown), an optical flow subtracting portion (not shown), 
and an optical flow adding portion (not shown). 
[0059] The first optical flow detecting portion is adapt- 
ed to detect first optical flow segments indicative of loci 
of the one or more moving objects apparently moved in 
the moving picture sequence in a time interval starting 
from the second time point to the first time point on the 
basis of the first picture frame inputted by the picture 
inputting section 410 and the second picture frame 
stored in the picture storage section 420. The second 
optical flow detecting portion is adapted to detect sec- 
ond optical flow segments indicative of loci of the one 
or more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the third 
time point to the first time point on the basis of the first 
picture frame inputted by the picture inputting section 
41 0 and the third picture frame stored in the picture stor- 
age section 420. 

[0060] More specifically, the first optical flow detecting 
portion includes: a first picture frame dividing unit (not 
shown), a second picture frame dividing unit (not 
shown), a comparing unit (not shown), and an optical 
flow detecting unit (not shown). 
[0061 ] The first picture frame dividing unit is adapted 
to divide the first picture frame stored in the picture stor- 
age section 420 into a plurality of blocks each containing 
at least one picture element. The second picture frame 
dividing unit is adapted to divide the second picture 
frame stored in the picture storage section 420 into a 
plurality of blocks each containing at least one picture 
element. The comparing unit is adapted to sequentially 
compare each of the blocks of the second picture frame 
divided by the second picture frame dividing unit with 
blocks of the first picture frame divided by the first picture 
frame dividing unit to select a block of the first picture 
frame apparently placed in the vicinity of the block of the 
second picture frame and having the highest degree of 
similarity with the block of the second picture frame from 
among the blocks of the first picture frame apparently 
placed in the vicinity of the block of the second picture 
frame. The optical flow detecting unit is adapted to de- 
tect optical flow segments each starting from a block of 
the second picture frame divided by the second picture 
frame dividing unit to a block of the first picture frame 
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selected by the comparing unit apparently placed in the 
vicinity of the block of the second picture frame and hav- 
ing the highest degree of similarity with the block of the 
second picture frame from among the blocks of the first 
picture frame apparently placed in the vicinity of the s 
block of the second picture frame. 
[0062] The description hereinlater is directed to the 
operation of the first optical flow detecting portion. 
[0063] The first picture frame dividing unit is operated 
to divide the first picture frame stored in the picture stor- 10 
age section 420 into a plurality of blocks each containing 
at least one picture element. The second picture frame 
dividing unit is operated to divide the second picture 
frame stored in the picture storage section 420 into a 
plurality of blocks each containing at least one picture is 
element. The comparing unit is operated to sequentially 
compare each of the blocks of the second picture frame 
divided by the second picture frame dividing unit with 
blocks of the first pictu re frame divided by the first picture 
frame dividing unit to select a block of the first picture 20 
frame apparently placed in the vicinity of the block of the 
second picture frame and having the highest degree of 
similarity with the block of the second picture frame from 
among the blocks of the first picture frame apparently 
placed in the vicinity of the block of the second picture 25 
frame. The optical flow detecting unit is operated to de- 
tect optical flow segments each starting from a block of 
the second picture frame divided by the second picture 
frame dividing unit to a block of the first picture frame 
selected, by the comparing unit apparently placed in the 30 
vicinity of the block of the second picture frame and hav- 
ing the highest degree of similarity with the block of the 
second picture frame from among the blocks of the first 
picture frame apparently placed in the vicinity of the 
block of the second picture frame. 35 
[0064] The false optical flow detecting portion is 
adapted to detect third optical flow segments indicative 
of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 
starting from the fourth time point to the third time point 40 
on the basis of the third picture frame stored in the pic- 
ture storage section 420 and the fourth picture frame 
stored in the picture storage section 420. The optical 
flow subtracting portion is adapted to subtract the third 
optical flow segments detected by the false optical flow 45 
detecting portion from the second optical flow segments 
detected in the second optical flow detecting portion to 
generate filtered optical flow segments indicative of loci 
of the one or more moving objects apparently moved in 
the moving picture sequence in the time interval starting so 
from the third time point to the first time point. 
[0065] The optical flow adding portion is adapted to 
add the filtered optical flow segments generated by the 
optical flow subtracting portion to the first optical flow 
segments detected by the first optical flow detecting por- S5 
tion to generate integrated optical flow segments indic- 
ative of loci of the one or more moving objects appar- 
ently moved in the moving picture sequence in the time 



interval starting from the third time point to the first time 
point. 

[0066] The third embodiment of the moving object de- 
tecting apparatus 400 further comprises a moving object 
detecting section 440. The moving object detecting sec- 
tion 440 is adapted to detect the moving objects on the 
basis of the integrated optical flow segments detected 
by the optical flow detecting section 430 by carrying out 
the step S320 of the second embodiment of the moving 
object detecting method as described earlier. The mov- 
ing object detecting section 440 includes: a moving ob- 
ject candidate area estimating portion 441 and a moving 
object area estimating portion 442. 
[0067] The moving object candidate area estimating 
portion 441 is adapted to detect neighboring areas 
neighboring to optical flow generating areas in which the 
integrated optical flow segments are detected by the op- 
tical flow detecting section 430 to obtain neighboring ar- 
ea information, and estimate moving object candidate 
areas in which the moving objects are supposed to be 
present on the basis of the neighboring area information 
detected thus detected to obtain moving object candi- 
date area information. The moving object area estimat- 
ing portion 442 is adapted to estimate moving object ar- 
eas in which the moving objects are present on the basis 
of the moving object candidate area information thus ob- 
tained by the moving object candidate area estimating 
portion 441 to detect the moving objects. 
[0068] The operation of the third embodiment of the 
moving object detecting apparatus will be described 
hereinlater. 

[0069] The picture inputting section 410 is operated 
to input the moving picture sequence including the first 
picture frame, the second picture frame, the third picture 
frame, and the fourth picture frame. The picture storage 
section 420 is operated to store the moving picture se- 
quence including the first picture frame, the second pic- 
ture frame, the third picture frame, and the fourth picture 
frame inputted by the picture inputting section 410. 
[0070] The optical flow detecting section 430 is oper- 
ated to detect integrated optical flow segments on the 
basis of the first picture frame inputted by the picture 
inputting section 410 and the second picture frame, the 
third picture frame, and the fourth picture frame stored 
in the picture storage section 420 by performing the first 
embodiment of the moving object detecting method as 
described earlier. The detailed description about the op- 
eration of the optical flow detecting section 430 will be 
therefore omitted to avoid tedious repetition. 
[0071 ] The moving object detecting section 440 is op- 
erated to detect the moving objects on the basis of the 
integrated optical flow segments detected by the optical 
flow detecting section 430 by carrying out the step S320 
of the second embodiment of the moving object detect- 
ing method as described earlier. The operation of the 
moving object detecting section 440 will be omitted from 
description to avoid tedious repetition. 
[0072] In the third embodiment of the moving object 
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detecting apparatus according to the present invention, 
the first optical flow detecting portion may detect first 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the sec- s 
ond time point to the first time point on the basis of the 
first picture frame inputted by the picture inputting sec- 
tion 410 and the second picture frame stored in the pic- 
ture storage section 420 and the second optical flow de- 
tecting portion may detect second optical flow segments 10 
indicative of loci of the one or more moving objects ap- 
parently moved in the moving picture sequence in a time 
interval starting from the third time point to the first time 
point on the basis of the first picture frame inputted by 
the picture inputting section 410 and the third picture 15 
frame stored in the picture storage section 420. 
[0073] Alternatively, the first optical flow detecting 
portion may detect first optical flow segments indicative 
of loci of the one or more moving objects apparently 
moved in the moving picture sequence in a time interval 20 
starting from the second time point to the first time point 
on the basis of the first picture frame and the second 
picture frame stored in the picture storage section 420. 
Furthermore, the second optical flow detecting portion 
may detect second optical flow segments indicative of 25 
loci of the one or more moving objects apparently moved 
in the moving picture sequence in a time interval starting 
from the third time point to the first time point on the basis 
of the first picture frame and the third picture frame 
stored in the picture storage section 420. 30 
[0074] From the foregoing description, it is to be un- 
derstood that the third embodiment of the moving object 
detecting apparatus according to the present invention 
can detect optical flow segments indicative of loci of one 
or more moving objects apparently moved at variable 35 
moving speeds in a moving picture sequence including 
a plurality of picture frames each indicating the one or 
more moving objects at a time point to ensure that the 
moving objects are correctly detected. 
[0075] Furthermore, the third embodiment of the mov- 40 
ing object detecting apparatus according to the present 
invention can accurately detect optical flow segments 
indicative of loci of one or more moving objects appar- 
ently moved at variable moving speeds in a moving pic- 
ture sequence including a plurality of picture frames 45 
each indicating the one or more moving objects at a time 
point by eliminating false optical flow segments which 
inaccurately indicate loci of a moving object apparently 
moved at a moving speed so high that the moving object 
apparently moved to an undetectable area away from so 
the detectable area in which optical flow segments can 
be properly detected and generated to ensure that the 
moving objects are correctly detected. 
[0076] Referring to FIG. 5 of the drawings, there is 
shown a fourth preferred embodiment of the moving ob- 55 
ject detecting method according to the present inven- 
tion. The fourth embodiment of the moving object de- 
tecting method is similar to the second embodiment of 



the moving object detecting method except for the fact 
that the fourth embodiment of the moving object detect- 
ing method further comprises the steps of storing the 
first picture frame, the second picture frame, the third 
picture frame, and the fourth picture frame, tracking the 
moving objects and controlling the time interval between 
the first time point and the second time point, the time 
interval between the second time point and the third time 
point, and the time interval between the third time point 
and the fourth time point, and the number of the inte- 
grated optical flow segments on the basis of the moving 
object information. 

[0077] The fourth embodiment of the moving object 
detecting method according to the present invention will 
be described with reference to FIG. 5. 
[0078] In the step S510, the first picture frame is in- 
putted and stored. The step S510 goes forward to the 
step S521 , in which the second picture frame is inputted 
and stored. The step S521 goes forward to the step 
S131 , in which first optical flow segments indicative of 
loci of the one or more moving objects apparently moved 
in the moving picture sequence in a time interval starting 
from the second time point to the first time point is de- 
tected on the basis of the first picture frame inputted and 
stored in the step S510 and the second picture frame 
inputted and stored the step S521 . In the step S522, the 
second picture frame is inputted and stored. The step 
S522 goes forward to the step S132, in which second 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
picture sequence in a time interval starting from the third 
time point to the first time point is detected on the basis 
of the first picture frame inputted and stored the step 
S51 0 and the third picture frame inputted and stored the 
step S522 in a similar manner as described above. 
[0079] In the step S523, the fourth picture frame is in- 
putted and stored. The step S523 goes forward to the 
step S133 in which the third optical flow segments in- 
dicative of loci of the one or more moving objects appar- 
ently moved in the moving picture sequence in a time 
interval starting from the fourth time point to the third 
time point is detected on the basis of the third picture 
frame inputted and stored in the step S522 and the 
fourth picture frame inputted and stored in the step 
S523. Hie step S133 goes forward to the step S142 in 
which the third optical flow segments detected in the 
step S133 from the second optical flow segments de- 
tected in the step S132 to generate filtered optical flow 
segments indicative of loci of the one or more moving 
objects apparently moved in the moving picture se- 
quence in the time interval starting from the third time 
point to the first time point. 

[0080] The step S1 42 goes forward to the step S1 50 
in which the filtered optical flow segments generated in 
the step S142 are added to the first optical flow seg- 
ments detected in the step S131 to generate integrated 
optical flow segments indicative of loci of the one or 
more moving objects apparently moved in the moving 
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picture sequence in the time interval starting from the 
third time point to the first time point, which contain false 
less optical flow segments. The step S1 50 goes forward 
to the step S320, in which the moving objects are de- 
tected on the basis of the integrated optical flow seg- s 
ments generated in the step S1 50. The step S1 50 goes 
forward to the step S320 in which the moving objects 
are detected on the basis of the integrated optical flow 
segments generated in the step S150. 
[0081] The fourth embodiment of the moving object 10 
detecting method according to the present invention fur- 
ther comprises the step 530 of tracking the moving ob- 
jects and the step S540 of controlling the time interval 
between the first time point and the second time point, 
the time interval between the second time point and the *5 
third time point, and the time interval between the third 
time point and the fourth time point, and the number of 
the integrated optical flow segments on the basis of the 
moving object information. 

[0082] The step S320 goes forward to the step S530, 20 
in which the moving objects in the moving picture se- 
quence detected in the step S320 are tracked on the 
basis of the moving object candidate area information 
in the moving picture sequence accumulatively obtained 
in the second step of step S320 to estimate moving ob- 25 
ject information including, for example but not limited to, 
the number of the moving objects and the amount of ap- 
parent motion made by the moving objects. Then, the 
step S530 goes forward to the step S540, in which the 
time interval between the first time point and th e second 30 
time point, the time interval between the second time 
point and the third time point, and the time interval be- 
tween the third time point and the fourth time point, and 
the number of the integrated optical flow segments are 
controlled on the basis of the moving object information 35 
estimated in the step S530. 

[0083] The step S540 goes forward to the step S550, 
in which it is judged whetherthe moving object detecting 
process will be completed or not. If it is judged by, for 
example but not limited to, an operator, that all the mov- *o 
ing objects are detected, the step S540 goes to END. 
Otherwise, the step S540 goes back to the step S510 
and the moving object detecting process will continue 
with a first picture frame, a second picture frame, a third 
picture frame, and a fourth picture frame newly inputted. 
[0084] More specifically, in the step S320, firstly, 
neighboring areas neighboring to optical flow generat- 
ing areas in which the integrated optical flow segments 
are detected in the step S150 are detected to obtain 
neighboring area information in the moving picture se- so 
quence, secondly, moving object candidate areas in the 
moving picture sequence in which the moving objects 
are supposed to be present are estimated on the basis 
of the neighboring area information thus obtained to ob- 
tain moving object candidate area information in the S5 
moving picture sequence; and thirdly, the moving ob- 
jects are detected on the basis of the moving object can- 
didate area information thus obtained. The steps S510 



to S540 are repeated until it is judged that the moving 
object detecting process will be completed. This means 
that the steps S51 0 to S320 are repeated, and the step 
S320 goes forward to the step S530, in which moving 
object candidate area information obtained in the sec- 
ond step of the step S320, and the moving objects de- 
tected in the third step of the step S320 are accumula- 
tively stored, and then, the moving object candidate ar- 
eas apparently moved in the moving picture frame se- 
quence are compared on the basis of the moving object 
candidate area information accumulatively previously 
stored to track the moving objects detected in the third 
step of the step S320 to estimate moving object infor- 
mation about the moving objects. Then, the step S530 
goes forward to the step S540, in which the time interval 
between the first time point and the second time point, 
the time interval between the second time point and the 
third time point, and the time interval between the third 
time point and the fourth time point, and the number of 
the integrated optical flow segments are controlled on 
the basis of the moving object information estimated in 
the step S530. 

[0085] The fourth embodiment of the moving object 
detecting method thus described can adaptively and ef- 
fectively change the time interval between the first time 
point and the second time point, the time interval be- 
tween the second time point and the third time point, 
and the time interval between the third time point and 
the fourth time point, and the number of the integrated 
optical flow segments are controlled in accordance with 
the apparent motion of the moving object. This means 
that the fourth embodiment of the moving object detect- 
ing method can, for example, shorten the time interval 
starting from a past time point to a current time point to 
detect and generate optical flow segments indicative of 
apparent motion of a moving object moving at a high 
moving speed so that the moving object apparently 
moved in the moving picture sequence would remain in 
the detectable area, i.e., the blocks of the current picture 
frame at the current time point apparently placed in the 
vicinity of the block of the past picture frame at the past 
time point by a predetermined number of blocks or less 
while, on the other hand, that the fourth embodiment of 
the moving object detecting method can lengthen the 
time interval starting from a past time point to a current 
time point to detect and generate optical flow segments 
indicative of apparent motion of a moving object moving 
at a low moving speed so that the optical flow segments 
indicative of the apparent motion of the moving object 
apparently moved in the time interval would be detected 
and generated. This leads to the fact that the fourth em- 
bodiment of the moving object detecting method can 
stably detect optical flow segments to ensure that the 
moving objects are correctly detected. 
[0086] From the foregoing description, it is to be un- 
derstood that the fourth embodiment of the moving ob- 
ject detecting method according to the present invention 
can detect optical flow segments indicative of loci of one 
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or more moving objects apparently moved at variable 
moving speeds in a moving picture sequence including 
a plurality of picture frames each indicating the one or 
more moving objects at a time point to ensure that the 
moving objects are correctly detected. s 
[0087] Furthermore, the fourth embodiment of the 
moving object detecting method according to the 
present invention can accurately detect optical flow seg- 
ments indicative of loci of one or more moving objects 
apparently moved at variable moving speeds in a mov- io 
ing picture sequence including a plurality of picture 
frames each indicating the one or more moving objects 
at a time point by eliminating false optical flow segments 
which inaccurately indicate loci of a moving object ap- 
parently moved at a moving speed so high that the mov- * 5 
ing object apparently moved to an undetectable area 
away from the detectable area in which optical flow seg- 
ments can be property detected and generated to en- 
sure that the moving objects are correctly detected. 
[0088] Referring to FIG. 6 of the drawings, there is 20 
shown a fifth preferred embodiment of the moving object 
detecting apparatus 600 according to the present inven- 
tion. The fifth embodiment of the moving object detect- 
ing apparatus 600 is similar to the third embodiment of 
the moving object detecting apparatus 400 except for 25 
the fact that the fifth embodiment of the moving object 
detecting apparatus 600 further comprises a moving ob- 
ject tracking section 650 for tracking the moving objects, 
a moving object information storing section 660 for ac- 
cumulatively storing the moving object information, and 30 
an optical flow detecting controlling section 670 for con- 
trolling the time interval between the first time point and 
the second time point, the time interval between the sec- 
ond time point and the third time point, and the time in- 
terval between the third time point and the fourth time 35 
point, and the number of the integrated optical flow seg- 
ments on the basis of the moving object information 
stored in the moving object information storing section 
660. The same constitutional elements are simply rep- 
resented by the same reference numerals as those of 40 
the third embodiment of the moving object detecting ap- 
paratus 400, and will be thus omitted from detailed de- 
scription for avoiding tedious repetition. 
[0089] The fifth embodiment of the moving object de- 
tecting apparatus 600 will be described with reference 45 
to FIG. 6. The fifth embodiment of the moving object de- 
tecting apparatus 600 is shown in FIG. 6 as comprising: 
a camera unit 500, a picture inputting section 410, a pic- 
ture storage section 420, and an optical flow detecting 
section 430, a moving object detecting section 440, a so 
moving object tracking section 650, a moving object in- 
formation storing section 660, and an optical flow de- 
tecting controlling section 670. 

[0090] The camera unit 500 is adapted to output mov- 
ing picture sequence including a first picture frame indi- 55 
eating the one or more moving objects at a first time 
point, a second picture frame indicating the one or more 
moving objects at a second time point, a third picture 



frame indicating the one or more moving objects at a 
third time point, and a fourth picture frame indicating the 
one or more moving objects at a fourth time point in late 
order. 

[0091] The picture inputting section 41 0 is adapted to 
input the moving picture sequence including the first pic- 
ture frame, the second picture frame, the third picture 
. frame, and the fourth picture frame. The picture storage 
section 420 is adapted to store the moving picture se- 
quence including the first picture frame, the second pic- 
ture frame, the third picture frame, and the fourth picture 
frame inputted by the picture inputting section 410. 
[0092] The optical flow detecting section 430 is adapt- 
ed to detect integrated optical flow segments on the ba- 
sis of the first picture frame inputted by the picture in- 
putting section 410 and the second picture frame, the 
third picture frame, and the fourth picture frame stored 
in the picture storage section 420 by performing the first 
embodiment of the moving object detecting method as 
described earlier. 

[0093] The moving object detecting section 440 is 
adapted to detect the moving objects on the basis of the 
integrated optical flow segments detected by the optical 
flow detecting section 430. More specifically, the moving 
object detecting section 440 includes: a moving object 
candidate area estimating portion 441 and a moving ob- 
ject area estimating portion 442. The moving object can- 
didate area estimating portion 441 is adapted to detect 
neighboring areas neighboring to optical flow generat- 
ing areas in which the integrated optical flow segments 
are detected by the optical flow detecting section 430 to 
obtain neighboring area information, and estimate mov- 
ing object candidate areas in which the moving objects 
are supposed to be present on the basis of the neigh- 
boring area information detected thus detected to obtain 
moving object candidate area information. The moving 
object area estimating portion 442 is adapted to esti- 
mate moving object areas in which the moving objects 
are present on the basis of the moving object candidate 
area information thus obtained by the moving object 
candidate area estimating portion 441 to detect the mov- 
ing objects. 

[0094] The moving object tracking section 650 is 
adapted to track the moving objects detected by the 
moving object detecting section 440 on the basis of the 
moving object candidate area information in the moving 
picture sequence obtained by the moving object candi- 
date area estimating portion 441 to estimate moving ob- 
ject information including, for example but not limited to, 
the number of the moving objects and the amount of ap- 
parent motion made by the moving objects. 
[0095] The optical flow detecting controlling section 
670 is adapted to control the time interval between the 
first time point and the second time point, the time inter- 
val between the second time point and the third time 
point, and the time interval between the third time point 
and the fourth time point, and the number of the inte- 
grated optical flow segments on the basis of the moving 
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object information estimated by the moving object track- 
ing section 650. 

[0096] The operation of the fifth embodiment of the 
moving object detecting apparatus 600 will be described 
hereinlater. 

[0097] The camera unit 500 is operated to output 
moving picture sequence including a first picture frame 
indicating the one or more moving objects at a first time 
point, a second picture frame indicating the one or more 
moving objects at a second time point, a third picture 
frame indicating the one or more moving objects at a 
third time point, and a fourth picture frame indicating the 
one or more moving objects at a fourth time point in late 
order. 

[0098] The picture inputting section 410 is operated 
to input the moving picture sequence including the first 
picture frame, the second picture frame, the third picture 
frame, and the fourth picture frame. The picture storage 
section 420 is operated to store the moving picture se- 
quence including the first picture frame, the second pic- 
ture frame, the third picture frame, and the fourth picture 
frame inputted by the picture inputting section 410. 
[0099] The optical flow detecting section 430 is oper- 
ated to detect integrated optical flow segments on the 
basis of the first picture frame inputted by the picture 
inputting section 41 0 and the second picture frame, the 
third picture frame, and the fourth picture frame stored 
in the picture storage section 420 by performing the first 
embodiment of the moving object detecting method as 
described earlier. 

[01 00] The moving object detecting section 440 is op- 
erated to detect the moving objects on the basis of the 
integrated optical flow segments detected by the optical 
flow detecting section 430. More specifically, the moving 
object detecting section 440 includes: a moving object 
candidate area estimating portion 441 and a moving ob- 
ject area estimating portion 442. The moving object can- 
didate area estimating portion 441 is operated to detect 
neighboring areas neighboring to optical flow generat- 
ing areas in which the integrated optical flow segments 
are detected by the optical flow detecting section 430 to 
obtain neighboring area information, and estimate mov- 
ing object candidate areas in which the moving objects 
are supposed to be present on the basis of the neigh- 
boring area information detected thus detected to obtain 
moving object candidate area information. The moving 
object area estimating portion 442 is operated to esti- 
mate moving object areas in which the moving objects 
are present on the basis of the moving object candidate 
area information thus obtained by the moving object 
candidate area estimating portion 441 to detect the mov- 
ing objects. 

[0101] The moving object tracking section 650 is op- 
erated to track the moving objects detected by the mov- 
ing object detecting section 440 on the basis of the mov- 
ing object candidate area information in the moving pic- 
ture sequence obtained by the moving object candidate 
area estimating portion 441 to estimate moving object 



information including, for example but not limited to, the 
number of the moving objects and the amount of appar- 
ent motion made by the moving objects. 
[0102] The optical flow detecting controlling section 

5 670 is operated to control the time interval between the 
first time point and the second time point, the time inter- 
val between the second time point and the third time 
point, and the time interval between the third time point 
and the fourth time point, and the number of the inte- 

w grated optical flow segments on the basis of the moving 
object information stored in the moving object informa- 
tion storing section 660. 

[01 03] Furthermore, the picture inputting section 41 0 
may, for example, sequentially input the moving picture 
'5 sequence including the first picture frame, the second 
picture frame, the third picture frame, and the fourth pic- 
ture frame. The description hereinlater will be directed 
to the operation of the fifth embodiment of the moving 
object detecting apparatus 600 sequentially inputting 

20 the moving picture sequence including the first picture 
frame, the second picture frame, the third picture frame, 
and the fourth picture frame. The picture inputting sec- 
tion 41 0 is operative to sequentially input the moving pic- 
ture sequence including the first picture frame, the sec- 

25 ond picture frame, the third picture frame, and the fourth 
picture frame. The picture storage section 420 is oper- 
ative to sequentially store the moving picture sequence 
including the first picture frame, the second picture 
frame, the third picture frame, and the fourth picture 

30 frame inputted by the picture inputting section 41 0. 
[01 04] The optical flow detecting section 430 is oper- 
ative to sequentially detect integrated optical flow seg- 
ments on the basis of the first picture frame inputted by 
the picture inputting section 41 0 and the second picture 

35 frame, the third picture frame, and the fourth picture 
frame stored in the picture storage section 420. The 
moving object detecting section 440 is operative to se- 
quentially detecting the moving objects on the basis of 
the integrated optical flow segments detected by the op- 

40 tical flow detecting section 430. The moving object in- 
formation storing section 660 is operative to accumula- 
tively store the moving object candidate area informa- 
tion in the moving picture sequence sequentially ob- 
tained by the moving object candidate area estimating 

45 portion 441 of the moving object detecting section 440. 
[0105] The moving object tracking section 650 is op- 
erative to track the moving objects detected by the mov- 
ing object detecting section 440 on the basis of the mov- 
ing object candidate area information in the moving pic- 

50 ture sequence accumulatively stored in the moving ob- 
ject information storing section 660 to estimate moving 
object information including the number of the moving 
objects and the amount of apparent motion made by the 
moving objects. 

55 [0106] Alternatively, the moving object tracking sec- 
tion 650 may includes a moving object information stor- 
ing portion (not shown), and a comparing portion (not 
shown). 



16 



31 EP 1 282 

[0107] The moving object information storing portion 
is operative to accumulatively store the moving objects 
detected by the moving object area estimating portion 
442 and the moving object candidate area information 
in the moving picture sequence obtained by the moving 5 
object candidate area estimating portion 441 . The com- 
paring portion is operative to compare the moving object 
candidate areas apparently moved in the moving picture 
frame sequence on the basis of the moving object can- 
didate area information accumulatively stored in the 10 
moving object information storing portion to track the 
moving objects detected by the moving object area es- 
timating portion 442 to estimate moving object informa- 
tion about the moving objects. 

[01 08] The fifth embodiment of the moving object de- is 
tecting apparatus 600 thus constructed can adaptiveiy 
and effectively change the time interval between the first 
time point and the second time point, the time interval 
between the second time point and the third time point, 
and the time interval between the third time point and 20 
the fourth time point, and the number of the integrated 
optical flow segments are controlled in accordance with 
the apparent motion of the moving object. 
[0109] From the foregoing description, it is to be un- 
derstood that the fifth embodiment of the moving object 25 
detecting apparatus according to the present invention 
can detect optical flow segments indicative of loci of one 
or more moving objects apparently moved at variable 
moving speeds in a moving picture sequence including 
a plurality of picture frames each indicating the one or 30 
more moving objects at a time point to ensure that the 
moving objects are correctly detected. 
[01 1 0] Furthermore, the fifth embodiment of the mov- 
ing object detecting apparatus according to the present 
invention can accurately detect optical flow segments 35 
indicative of loci of one or more moving objects appar- 
ently moved at variable moving speeds in a moving pic- 
ture sequence including a plurality of picture frames 
each indicating the one or more moving objects at a time 
point by eliminating false optical flow segments which 40 
inaccurately indicate loci of a moving object apparently 
moved at a moving speed so high that the moving object 
apparently moved to an undetectable area away from 
the detectable area in which optical flow segments can 
be property detected and generated to ensure that the 45 
moving objects are correctly detected. 
[0111] Referring to FIG. 7 of the drawings, there is 
shown a sixth preferred embodiment of the moving ob- 
ject detecting apparatus 700 according to the present 
invention. The elements and the parts of sixth embodi- so 
ment of the moving object detecting apparatus 700 is 
entirely the same as those of the fifth embodiment of the 
moving object detecting apparatus 600 except for the 
fact that the camera unit 500, the picture inputting sec- 
tion 41 0, and the picture storage section 420 of the fifth 55 
embodiment of the moving object detecting apparatus 
600 are replaced with an infrared camera unit 701, a 
heat distribution picture inputting section 710, and a 
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heat distribution picture storage section 720. 
[01 12] Therefore, only the elements and parts of the 
sixth embodiment of the moving object detecting appa- 
ratus 700 different from the those of the fifth embodi- 
ment of the moving object detecting apparatus 600 will 
be described in detail hereinlater, but the elements and 
the parts of the sixth embodiment of the moving object 
detecting apparatus 700 entirely the same as those of 
the fifth embodiment of the moving object detecting ap- 
paratus 600 bear the same reference numerals as those 
of the fifth embodiment of the moving object detecting 
apparatus 600, and will be thus omitted from description 
for avoiding tedious repetition. 
[01 1 3] The sixth embodiment of the moving object de- 
tecting apparatus 700, further comprises an infrared 
camera unit 701 . The infrared camera unit 701 is adapt- 
ed to output a moving picture sequence of a heat distri- 
bution picture sequence including a plurality of picture 
frames including a first picture frame indicating the one 
or more moving objects at a first time point, a second 
picture frame indicating the one or more moving objects 
at a second time point, a third picture frame indicating 
the one or more moving objects at a third time point, and 
a fourth picture frame indicating the one or more moving 
objects at a fourth time point in late order. The infrared 
camera unit 701 may be, for example but not limited to, 
a far-infrared camera unit, which permits to convert op- 
tical moving picture information into digital moving pic- 
ture information even in darkness to output the moving 
picture sequence of a heat distribution picture se- 
quence. 

[0114] In the sixth embodiment of the moving object 
detecting apparatus 700, the picture inputting section 
410 is replaced with a heat distribution picture inputting 
section 710; and the picture storage section 420 is re- 
placed with a heat distribution picture storage section 
720. The heat distribution picture inputting section 710 
is adapted to input the first picture frame, the second 
picture frame, the third frame picture, and the fourth 
frame picture forming part of the moving picture se- 
quence of a heat distribution picture sequence outputted 
by the infrared camera unit 701 . The heat distribution 
picture storage section 720 is adapted to store the first 
picture frame, the second picture frame, the third frame 
picture, and the fourth frame picture forming part of the 
moving picture sequence of a heat distribution picture 
sequence inputted by the heat distribution picture input- 
ting section 710. 

[0115] The elements and the parts of the sixth embod- 
iment of the moving object detecting apparatus 700 is 
entirely the same as those of the fifth embodiment of the 
moving object detecting apparatus 600 except for the 
fact that the camera unit 500, the picture inputting sec- 
tion 41 0, and the picture storage section 420 of the fifth 
embodiment of the moving object detecting apparatus 
600 are replaced with an infrared camera unit 701 , a 
heat distribution picture inputting section 710, and a 
heat distribution picture storage section 720. 



17 



33 EP 1 282 077 A2 34 



indicating said one or more moving objects at a third 
time point, and a fourth picture frame indicating said 
one or more moving objects at a fourth time point in 
late order, said method comprising the steps of: 

5 

a) inputting said first picture frame; 

b) inputting said second picture frame; 

c) inputting said third picture frame; 

d) inputting said fourth picture frame; 

10 e) detecting first optical flow segments indica- 

tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said sec- 
ond time point to said first time point on the ba- 

15 sis of said first picture frame inputted in said 

step (a) and said second picture frame inputted 
in said step (b); 

f) detecting second optical flow segments indic- 
ative of loci of said one or more moving objects 

20 apparently moved in said moving picture se- 

quence in a time interval starting from said third 
time point to said first time point on the basis of 
said first picture frame inputted in said step (a) 
and said third picture frame inputted in said 

25 step (c); 

g) detecting third optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said 

30 fourth time point to said third time point on the 

basis of said third picture frame inputted in said 
step (c) and said fourth picture frame inputted 
in said step (d); 

h) subtracting said third optical flow segments 
35 detected in said step (g) from said second op- 
tical flow segments detected in said step (f) to 
generate filtered optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 

40 quence in said time interval starting from said 

third time point to said first time point; and 

i) adding said filtered optical flow segments 
generated in said step (h) to said first optical 
flow segments detected in said step (e) to gen- 

45 erate integrated optical flow segments indica- 

tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point. 



[0116] From the foregoing description, it is to be un- 
derstood that the sixth embodiment of the moving object 
detecting apparatus 700 according to the present inven- 
tion can detect optical flow segments indicative of loci 
of one or more moving objects apparently moved at var- 
iable moving speeds in a moving picture sequence in- 
cluding a plurality of picture frames each indicating the 
one or more moving objects at a time point to ensure 
that the moving objects are correctly detected. 
[0117] Furthermore, the sixth embodiment of the 
moving object detecting apparatus 700 according to the 
present invention can accurately detect optical flow seg- 
ments indicative of loci of one or more moving objects 
apparently moved at variable moving speeds in a mov- 
ing picture sequence including a plurality of picture 
frames each indicating the one or more moving objects 
at a time point by eliminating false optical flow segments 
which inaccurately indicate loci of a moving object ap- 
parently moved at a moving speed so high that the mov- 
ing object apparently moved to an undetectable area 
away from the detectable area in which optical flow seg- 
ments can be properly detected and generated to en- 
sure that the moving objects are correctly detected. 
[01 1 8] The above described moving object detecting 
method is preferably implemented by software for pro- 
gramming a computer, typically the computer used to 
process the raw data obtained from a camera unit. Such 
program may be embodied in a program storage medi- 
um readable by machine, tangibly embodying a pro- 
gram of instructions executable by the machine to per- 
form a set of method steps necessary to implement the 
above described program. Such program storage me- 
dium may be, for example but not limited to, a floppy 
disk, a CD ROM disk, a magnetic tape medium, an in- 
ternal computer memory, or any other medium or device 
capable of storing information readable by the compu- 
ter. It may be a free standing program or it may be a 
portion of a larger program executable by the computer. 
[0119] Although the particular embodiment of the 
present invention has been shown and described, it will 
be obvious to those skilled in the art that various chang- 
es and modifications may be mode without departing 
from the spirit and scope of the present invention . 



Claims 

1 . A moving object detecting method of detecting op- 
tical flow segments indicative of loci of one or more 
moving objects apparently moved in a moving pic- 50 
ture sequence including a plurality of picture frames 
each indicating said one or more moving objects at 
a time point to ensure that said one or more moving 
objects are correctly detected, said picture frames 
including a first picture frame indicating said one or . 55 
more moving objects at a first time point, a second 
picture frame indicating said one or more moving 
objects at a second time point, a third picture frame 



2. A moving object detecting method as set forth in 
claim 1 , in which each of said picture frame includes 
a plurality of blocks, and said step (e) has the steps 
of: 

(e1) dividing said first picture frame inputted in 
said step (a) into a plurality of blocks each con- 
taining at least one picture element; 
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(e2) dividing said second picture frame inputted 
in said step (b) into a plurality of blocks each 
containing at least one picture element; 
(e3) sequentially comparing each of said blocks 
of said second picture frame divided in said 5 
step (e2) with blocks of said first picture frame 
divided in said step (e1 ) to select a block of said 
first picture frame apparently placed in the vi- 
cinity of said block of said second picture frame 
and having the highest degree of similarity with io 
said block of said second picture frame from 
among said blocks of said first picture frame ap- 
parently placed in the vicinity of said block of 
said second picture frame; and 
(e4) detecting optical flow segments each start- 15 
ing from a block of said second picture frame 
divided in said step (e2) to a block of said first 
picture frame selected in said step (e3) appar- 
ently placed in the vicinity of said block of said 
second picture frame and having the highest 20 
degree of similarity with said block of said sec- 
ond picture frame from among said blocks of 
said first picture frame apparently placed in the 
vicinity of said block of said second picture 
frame. 25 

3. A moving object detecting method as set forth in 
claim 1 , further comprising the steps of: 

j) detecting neighboring areas neighboring to 30 
optical flow generating areas in which said in- 
tegrated optical flow segments are detected in 
said step (i) to obtain neighboring area informa- 
tion; 

k) estimating moving object candidate areas in 35 
said moving picture sequence in which said 
moving objects are supposed to be present on 
the basis of said neighboring area information 
detected in said step (j) to obtain moving object 
candidate area information in said moving pic- *o 
ture sequence; and 

I) detecting said moving objects on the basis of 
said moving object candidate area information 
obtained in said step (k). 

45 

4. A moving object detecting method as set forth in 
claim 3, in which 

said moving object candidate area informa- 
tion includes information about the sizes, shapes, 
positions, picture elements, optical flow information so 
of the moving object candidate areas. 

5. A moving object detecting method of detecting op- 
tical flow segments indicative of loci of one or more 
moving objects apparently moved in a moving pic- ss 
ture sequence including a plurality of picture frames 
each indicating said one or more moving objects at 

a time point to ensure that said one or more moving 



objects are correctly detected, said picture frames 
including a first picture frame indicating said one or 
more moving objects at a first time point, a second 
picture frame indicating said one or more moving 
objects at a second time point, a third picture frame 
indicating said one or more moving objects at a third 
time point, and a fourth picture frame indicating said 
one or more moving objects at a fourth time point in 
late order, said method comprising the steps of: 

a2) inputting said first picture frame; 
b2) inputting said second picture frame; 
c2) inputting said third picture frame; 
d2) inputting said fourth picture frame; 
a3) storing said first picture frame; 
b3) storing said second picture frame; 
c3) storing said third picture frame; 
d3) storing said fourth picture frame; 
e) detecting first optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said sec- 
ond time point to said first time point on the ba- 
sis of said first picture frame stored in said step 
(a2) and said second picture frame stored in 
said step (b2); 

0 detecting second optical flow segments indic- 
ative of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said third 
time point to said first time point on the basis of 
said first picture frame stored in said step (a2) 
and said third picture frame stored in said step 
(c2); 

g) detecting third optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said 
fourth time point to said third time point on the 
basis of said third picture frame stored in said 
step (c2) and said fourth picture frame stored 
in said step (d2); 

h) subtracting said third optical flow segments 
detected in said step (g) from said second op- 
tical flow segments detected in said step (0 to 
generate filtered optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; 

i) adding said filtered optical flow segments 
generated in said step (h) to said first optical 
flow segments detected in said step (e) to gen- 
erate integrated optical flow segments indica- 
tive of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; 
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j) detecting neighboring areas neighboring to 
optical fiow generating areas in which said in- 
tegrated optical flow segments are detected in 
said step (i) to obtain neighboring area informa- 
tion in said moving picture sequence; 5 
k) estimating moving object candidate areas in 
said moving picture sequence in which said 
moving objects are supposed to be present on 
the basis of said neighboring area information 
detected in said step (j) to obtain moving object io 
candidate area information in said moving pic- 
ture sequence; and 

I) detecting said moving objects in said moving 
picture sequence on the basis of said moving 
object candidate area information obtained in is 
said step (k). 

A moving object detecting method as set forth in 
claim 5, further comprising the steps of: 

20 

m) repeating said steps (a2) to (1 ), and tracking 
said moving objects in said moving picture se- . 
quence detected in said step (1) on the basis 
of said moving object candidate area informa- 
tion in said moving picture sequence accumu- 25 
lativery obtained in said step (k) to estimate 
moving object information including the number 
of said moving objects and the amount of ap- 
parent motion made by said moving objects; 
and 30 
n) controlling the time interval between said first 
time point and said second time point, the time 
interval between said second time point and 
said third time point, and the time interval be- 
tween said third time point and said fourth time 35 
point, and the number of said integrated optical 
flow segments on the basis of said moving ob- 
ject information estimated in said step (m). 



number of said moving objects and the amount of 
apparent motion made by said moving objects 

9. A moving object detecting apparatus (400) for de- 
tecting optical flow segments indicative of loci of 
one or more moving objects apparently moved in a 
moving picture sequence including a plurality of pic- 
ture frames each indicating said one or more mov- 
ing objects at a time point to ensure that said one 
or more moving objects are correctly detected, said 
picture frames including a first picture frame indicat- 
ing said one or more moving objects at a first time 
point, a second picture frame indicating said one or 
more moving objects at a second time point, a third 
picture frame indicating said one or more moving 
objects at a third time point, and a fourth picture 
frame indicating said one or more moving objects 
at a fourth time point in late order, said moving ob- 
ject detecting apparatus (400) comprising: 

a picture inputting section (410) for inputting 
said moving picture sequence including said 
first picture frame, said second picture frame, 
said third picture frame, and said fourth picture 
frame; 

a picture storage section (420) for storing said 
moving picture sequence including said first 
picture frame, said second picture frame, said 
third picture frame, and said fourth picture 
frame inputted by said picture inputting section 
(410); and 

an optical flow detecting section (430) for de- 
tecting integrated optical flow segments on the 
basis of said first picture frame inputted by said 
picture inputting section (410) and said second 
picture frame, said third picture frame, and said 
fourth picture frame stored in said picture stor- 
age section (420), 



A moving object detecting method as set forth in 40 
claim 6, in which said step (m) has the steps of: 

(ml) accumulatively storing moving object can- 
didate area information obtained in said step 
(k); and said moving objects detected in said 45 
step(1) ;and 

(m2) comparing said moving object candidate 
areas apparently moved in said moving picture 
frame sequence on the basis of said moving ob- 
ject candidate area information accumulatively so 
stored in said step (ml) to track said moving 
objects detected in said step (1) to estimate 
moving object information about said moving 
objects. 

55 

A moving object detecting method as set forth in 
claim 6, in which 

said moving object information including the 



whereby said optical flow detecting section (430) in- 
cludes: 

a first optical flow detecting portion for detecting 
first optical flow segments indicative of loci of 
said one or more moving objects apparently 
moved in said moving picture sequence in a 
time interval starting from said second time 
point to said first time point on the basis of said 
first picture frame inputted by said picture input- 
ting section (410) and said second picture 
frame stored in said picture storage section 
(420); 

a second optical flow detecting portion for de- 
tecting second optical flow segments indicative 
of loci of said one or more moving objects ap- 
parently moved in said moving picture se- 
quence in a time interval starting from said third 
time point to said first time point on the basis of 
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said first picture frame inputted by said picture 
inputting section (410) and said third picture 
frame stored in said picture storage section 
(420); 

a false optical flow detecting portion for detect- 5 
ing third optical flow segments indicative of loci 
of said one or more moving objects apparently 
moved in said moving picture sequence in a 
time interval starting from said fourth time point 
to said third time point on the basis of said third io 
picture frame stored in said picture storage sec- 
tion (420) and said fourth picture frame stored 
in said picture storage section (420); 
an optical flow subtracting portion for subtract- 
ing said third optical flow segments detected by is 
said false optical flow detecting portion from 
said second optical flow segments detected by 
said second optical flow detecting portion to 
generate filtered optical flow segments indica- 
tive of loci of said one or more moving objects 20 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; and 
an optical flow adding portion for adding said 
filtered optical flow segments generated by said 25 
optical flow subtracting portion to said first op- 
tical flow segments detected by said first optical 
flow detecting portion to generate integrated 
optical flow segments indicative of loci of said 
one or more moving objects apparently moved 30 
in said moving picture sequence in said time 
interval starting from said third time point to said 
first time point. 

A moving object detecting apparatus (400) as set 35 
forth in claim 9, in which 

said first optical flow detecting portion in- 
cludes: 

a first picture frame dividing unit for dividing *o 
said first picture frame stored by said picture 
storage section (420) into a plurality of blocks 
each containing at least one picture element; 
a second picture frame dividing unit for dividing 
said second picture frame stored by said pic- 45 
ture storage section (420) into a plurality of 
blocks each containing at least one picture el- 
ement; 

a comparing unit for sequentially comparing 
each of said blocks of said second picture so 
frame divided by said second picture frame di- 
viding unit with blocks of said first picture frame 
divided by said first picture frame dividing unit 
to select a block of said first picture frame ap- 
parently placed in the vicinity of said block of 55 
said second picture frame and having the high- 
est degree of similarity with said block of said 
second picture frame from among said blocks 



of said first picture frame apparently placed in 
the vicinity of said block of said second picture 
frame; and 

an optical flow detecting unit for detecting opti- 
cal flow segments each starting from a block of 
said second picture frame divided by said sec- 
ond picture frame, dividing unit to a block of 
said first picture frame selected by said com- 
paring unit apparently placed in the vicinity of 
said block of said second picture frame and 
having the highest degree of similarity with said 
block of said second picture frame from among 
said blocks of said first picture frame apparently 
placed in the vicinity of said block of said sec- 
ond picture frame. 

11. A moving object detecting apparatus (500) as set 
forth in claim 9, further comprising a moving object 
detecting section (440) for detecting said moving 
objects on the basis of said integrated optical flow 
segments detected by said optical flow detecting 
section (430) whereby said moving object detecting 
section (440) includes: 

a moving object candidate area estimating por- 
tion (441) for detecting neighboring areas 
neighboring to optical flow generating areas in 
which said integrated optical flow segments are 
detected by said optical flow detecting section 
(430) to obtain neighboring area information, 
and estimating moving object candidate areas 
in which said moving objects are supposed to 
be present on the basis of said neighboring ar- 
ea information detected thus detected to obtain 
moving object candidate area information; and 
a moving object area estimating portion (442) 
for estimating moving object areas in which 
said moving objects are present on the basis of 
said moving object candidate area information 
in said moving picture sequence obtained by 
said moving object candidate area estimating 
portion (441) to detect said moving objects. 

12. A moving object detecting apparatus (500) as set 
forth in claim 11 , in which 

said moving object candidate area informa- 
tion includes information about the sizes, shapes, 
positions, picture elements, optical flow information 
of the moving object candidate areas. 

13. A moving object detecting apparatus (600) as set 
forth in claim 11 , further comprising: 

a moving object tracking section (650) for track- 
ing said moving objects detected by said mov- 
ing object area estimating portion (442) on the 
basis of said moving object candidate area in- 
formation in said moving picture sequence ob- 
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tained by said moving object candidate area es- 
timating portion (441 ) to estimate moving object 
information including the number of said mov- 
ing objects and the amount of apparent motion 
made by said moving objects; and s 
an optical flow detecting controlling section 
(670) for controlling the time interval between 
said first time point and said second time point, 
the time interval between said second time 
point and said third time point, and the time in- w 
terval between said third time point and said 
fourth time point, and the number of said inte- 
grated optical flow segments on the basis of 
said moving object information estimated by 
said moving object tracking section (650). is 

14. A moving object detecting apparatus (600) as set 
forth in claim 13, which further comprising: a moving 
object information storing section (660) for accumu- 
latively storing said moving object candidate area 20 
information in said moving picture sequence se- 
quentially obtained by said moving object candidate 
area estimating portion (441) of said moving object 
detecting section (440), in which 

said picture inputting section (410) is opera- 25 
tive to sequentially input said moving picture se- 
quence including said first picture frame, said sec- 
ond picture frame, said third picture frame, and said 
fourth picture frame; 

said picture storage section (420) is operative 30 
to sequentially store said moving picture sequence 
including said first picture frame, said second pic- 
ture frame, said third picture frame, and said fourth 
picture frame inputted by said picture inputting sec- 
tion (410); 35 

said optical flow detecting section (430) is op- 
erative to sequentially detect integrated optical flow 
segments on the basis of said first picture frame in- 
putted by said picture inputting section (410) and 
said second picture frame, said third picture frame, 40 
and said fourth picture frame stored in said picture 
storage section (420); and 

said moving object detecting section (440) is 
operative to sequentially detecting said moving ob- 
jects on the basis of said integrated optical flow seg- 45 
ments detected by said optical flow detecting sec- 
tion (430); and 

said moving object tracking section (650) is 
operative to track said moving objects detected by 
said moving object detecting section (440) on the so 
basis of said moving object candidate area inf orma- 
tion in said moving picture sequence accumulative- 
ly stored in said moving object information storing 
section (660) to estimate moving object information 
including the number of said moving objects and the ss 
amount of apparent motion made by said moving 
objects. 



15. A moving object detecting apparatus (600) as set 
forth in daim 13, in which 

said moving object tracking section (650) in- 
cludes: 

a moving object information storing portion for 
accumulatively storing said moving objects de- 
tected by said moving object area estimating 
portion (442) and said moving object candidate 
area information in said moving picture se- 
quence obtained by said moving object candi- 
date area estimating portion (441); and 
a comparing portion for comparing said moving 
object candidate areas apparently moved in 
said moving picture frame sequence on the ba- 
sis of said moving object candidate area infor- 
mation accumulatively stored in said moving 
object information storing portion to track said 
moving objects detected by said moving object 
area estimating portion (442) to estimate mov- 
ing object information about said moving ob- 
jects. 

1 6. A moving object detecting apparatus (500) as set 
forth in claim 13, in which 

said moving object information including the 
number of said moving objects and the amount of 
apparent motion made by said moving objects 

7. A moving object detecting apparatus (700) as set 
forth in claim 9, further comprising an infrared cam- 
era unit (701 ) for outputting a moving picture se- 
quence of a heat distribution picture sequence in- 
cluding a plurality of picture frames including a first 
picture frame indicating said. one or more moving 
objects at a first time point, a second picture frame 
indicating said one or more moving objects at a sec- 
ond time point, a third picture frame indicating said 
one or more moving objects at a third time point, 
and a fourth picture frame indicating said one or 
more moving objects at a fourth time point in late 
order in which 

said picture inputting section (41 0) is replaced 
with a heat distribution picture inputting section 
(71 0) for inputting said first picture frame, said sec- 
ond picture frame, said third frame picture, and said 
fourth frame picture forming part of said moving pic- 
ture sequence of a heat distribution picture se- 
quence outputted by said infrared camera unit 
(701); and 

said picture storage section (420) is replaced 
with a heat distribution picture storage section (720) 
for storing said first picture frame, said second pic- 
ture frame, said third frame picture, and said fourth 
frame picture forming part of said movin g picture se- 
quence of a heat distribution picture sequence in- 
putted by said heat distribution picture inputting 
section (710). 
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18. A moving object detecting program product com- 
prising a computer usable storage medium having 
computer readable code encoded herein for detect- 
ing optical flow segments indicative of loci of one or 
more moving objects apparently moved in a moving s 
picture sequence including a plurality of picture 
frames each indicating said one or more moving ob- 
jects at a time point to ensure that said one or more 
moving objects are correctly detected, said picture 
frames including a first picture frame indicating said 10 
one or more moving objects at a first time point, a 
second picture frame indicating said one or more 
moving objects at a second time point, a third pic- 
ture frame indicating said one or more moving ob- 
jects at a third time point, and a fourth picture frame 15 
indicating said one or more moving objects at a 
fourth time point in late order, said computer pro- 
gram product comprising: 

(a) computer readable program code for input- 20 
ting said first picture frame; 

(b) computer readable program code for input- 
ting said second picture frame; 

(c) computer readable program code for input- 
ting said third picture frame; 

(d) computer readable program code for input- 
ting said fourth picture frame; 

(e) computer readable program code for detect- 
ing first optical flow segments indicative of loci 
of said one or more moving objects apparently 
moved in said moving picture sequence in a 
time interval starting from said second time 
point to said first time point on the basis of said 
first picture frame inputted by said computer 
readable program code (a) and said second 
picture frame inputted by said computer read- 
able program code (b); 

(f) computer readable program code for detect- 
ing second optical flow segments indicative of 
loci of said one or more moving objects appar- 
ently moved in said moving picture sequence 
in a time interval starting from said third time 
point to said first time point on the basis of said 
first picture frame inputted by said computer 
readable program code (a) and said third pic- 
ture frame inputted by said computer readable 
program code (c); 

(g) detecting third optical flow segments indic- 
ative of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in a time interval starting from said 
fourth time point to said third time point on the 
basis of said third picture frame inputted by said 
computer readable program code (c) and said 
fourth picture frame inputted by said computer 
readable program code (d); 

(h) computer readable program code for sub- 
tracting said third optical flow segments detect- 



ed by said computer readable program code (g) 
from said second optical flow segments detect- 
ed by said computer readable program code (f) 
to generate filtered optical flow segments indic- 
ative of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; and 
(i) computer readable program code for adding 
said filtered optical flow segments generated by 
said computer readable program code (h) to 
said first optical flow segments detected by said 
computer readable program code (e) to gener- 
ate integrated optical flow segments indicative 
of loci of said one or more moving objects ap- 
parently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point. 

19. A moving object detecting program product as set 
forth in claim 1 8, in which each of said picture frame 
includes a plurality of blocks, and said computer 
readable program code (e) has: 



(e1 ) computer readable program code for divid- 
ing said first picture frame inputted by said com- 
puter readable program code (a) into a plurality 
of blocks each containing at least one picture 
element; 

(e2) computer readable program code for divid- 
ing said second picture frame inputted by said 
computer readable program code (b) into a plu- 
rality of blocks each containing at least one pic- 
ture element; 

(e3) computer readable program code for se- 
quentially comparing each of said blocks of said 
second picture frame divided by said computer 
readable program code (e2) with blocks of said 
first picture frame divided by said computer 
readable program code (e1 ) to select a block of 
said first picture frame apparently placed in the 
vicinity of said block of said second picture 
frame and having the highest degree of similar- 
ity with said block of said second picture frame 
from among said blocks of said first picture 
frame apparently placed in the vicinity of said 
block of said second picture frame; and 
(e4) computer readable program code for de- 
tecting optical flow segments each starting from 
a block of said second picture frame divided by 
said computer readable program code (e2) to 
a block of said first picture frame selected by 
said computer readable program code (e3) ap- 
parently placed in the vicinity of said block'of 
said second picture frame and having the high- 
est degree of similarity with said block of said 
second picture frame from among said blocks 
of said first picture frame apparently placed in 
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the vicinity of said block of said second picture 
frame. 

20. A moving object detecting program product as set 
forth in claim 18, further comprising: 

(j) computer readable program code for detect- 
ing neighboring areas neighboring to optical 
flow generating areas in which said integrated 
optical flow segments are detected by said 
computer readable program code (i) to obtain 
neighboring area information; 
(k) computer readable program code for esti- 
mating moving object candidate areas in said 
moving picture sequence in which said moving 
objects are supposed to be present on the basis 
of said neighboring area information detected 
by said computer readable program code (j) to 
obtain moving object candidate area informa- 
tion in said moving picture sequence; and 
(I) computer readable program code for detect- 
ing said moving objects on the basis of said 
moving object candidate area information ob- 
tained by said computer readable program 
code (k). 

21 . A moving object detecting program product as set 
forth in claim 20, in which 

said moving object candidate area informa- 
tion includes information about the sizes, shapes, 
positions, picture elements, optical flow information 
of the moving object candidate areas. 

22. A moving object detecting program product for de- 
tecting optical flow segments indicative of loci of 
one or more moving objects apparently moved in a 
moving picture sequence including a plurality of pic- 
ture frames each indicating said one or more mov- 
ing objects at a time point to ensure that said one 
or more moving objects are correctly detected, said 
picture frames including a first picture frame indicat- 
ing said one or more moving objects at a first time 
point, a second picture frame indicating said one or 
more moving objects at a second time point, a third 
picture frame indicating said one or more moving 
objects at a third time point, and a fourth picture 
frame indicating said one or more moving objects 
at a fourth time point in late order, said computer 
program product comprising: 

(a2) computer readable program code for input- 
ting said first picture frame; 
(b2) computer readable program code for input- 
ting said second picture frame; 
(c2) computer readable program code for input- 
ting said third picture frame; 
(d2) computer readable program code for input- 
ting said fourth picture frame; 
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(a3) computer readable program code for stor- 
ing said first picture frame; 
(b3) computer readable program code for stor- 
ing said second picture frame; 
(c3) computer readable program code for stor- 
ing said third picture frame; 
(d3) computer readable program code for stor- 
ing said fourth picture frame; 
(e1) computer readable program code for de- 
tecting first optical flow segments indicative of 
loci of said one or more moving objects appar- 
ently moved in said moving picture sequence 
in a time interval starting from said second time 
point to said first time point on the basis of said 
first picture frame stored by said computer 
readable program code (a2) and said second 
picture frame stored by said computer readable 
program code (b2); 

(f) computer readable program code for detect- 
ing second optical flow segments indicative of 
loci of said one or more moving objects appar- 
ently moved in said moving picture sequence 
in a time interval starting from said third time 
point to said first time point on the basis of said 
first picture frame stored by said computer 
readable program code (a2) and said third pic- 
ture frame stored by said computer readable 
program code (c2); 

(g) computer readable program code for detect- 
ing third optical flow segments indicative of loci 
of said one or more moving objects apparently 
moved in said moving picture sequence in a 
time interval starting from said fourth time point 
to said third time point on the basis of said third 
picture frame stored by said computer readable 
program code (c2) and said fourth picture frame 
stored by said computer readable program 
code (d2); 

(h) computer readable program code for sub- 
tracting said third optical flow segments detect- 
ed by said computer readable program code (g) 
from said second optical flow segments detect- 
ed by said computer readable program code (f) 
to generate filtered optical flow segments indic- 
ative of loci of said one or more moving objects 
apparently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; 

(i) computer readable program code for adding 
said filtered optical flow segments generated by 
said computer readable program code (h) to 
said first optical flow segments detected by said 
computer readable program code (e) to gener- 
ate integrated optical flow segments indicative 
of loci of said one or more moving objects ap- 
parently moved in said moving picture se- 
quence in said time interval starting from said 
third time point to said first time point; 



24 



47 



EP 1 282 077 A2 



48 



(D computer readable program code for detect- 
ing neighboring areas neighboring to optical 
flow generating areas in which said integrated 
optical flow segments are detected by said 
computer readable program code (i) to obtain 5 
neighboring area information in said moving 
picture sequence; 

(k) computer readable program code for esti- 
mating moving object candidate areas in said 
moving picture sequence in which said moving w 
objects are supposed to be present on the basis 
of said neighboring area information detected 
by said computer readable program code (j) to 
obtain moving object candidate area informa- 
tion in said moving picture sequence; and 15 
(I) computer readable program code for detect- 
ing said moving objects in said moving picture 
sequence on the basis of said moving object 
candidate area information obtained by said 
computer readable program code (k). 20 

23. A moving object detecting program product as set 
forth in claim 22, further comprising: 

(m) computer readable program code for re- 25 
peating said computer readable program codes 
(a2) to (1), and tracking said moving objects in 
said moving picture sequence detected by said 
computer readable program code (1 ) on the ba- 
sis of said moving object candidate area infor- 30 
mation in said moving picture sequence accu- 
mulatively obtained by said computer readable 
program code (k) to estimate moving object in- 
formation including the number of said moving 
objects and the amount of apparent motion 35 
made by said moving objects; and 
(n) computer readable program code for con- 
trolling the time interval between said first time 
point and said second time point, the time in- 
terval between said second time point and said 40 
third time point, and the time interval between 
said third time point and said fourth time point, 
and the number of said integrated optical flow 
segments on the basis of said moving object 
information estimated by said computer reada- 45 
ble program code (m). 

24. A moving object detecting program product as set 
forth in claim 23, in which said computer readable 
program code (m) has: so 

(ml) computer readable program code for ac- 
cumulatively storing moving object candidate 
area information obtained by said computer 
readable program code (k); and said moving 55 
objects detected by said computer readable 
program code (1);and 

(m2) computer readable program code for com- 



paring said moving object candidate areas ap- 
parently moved in said moving picture frame 
sequence on the basis of said moving object 
candidate area information accumulatively 
stored by said computer readable program 
code (ml ) to track said moving objects detect- 
ed by said computer readable program code (1 ) 
to estimate moving object information about 
said moving objects. 

25. A moving object detecting program product 
as set forth in claim 23, in which 

said moving object information including 
the number of said moving objects and the 
amount of apparent motion made by said mov- 
ing objects 
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